<![CDATA[Bluesky is down today.]]>Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".

Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.

I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)

]]>https://board.circlewithadot.net/topic/ec2f0b10-2f53-444a-879b-854064ba3acf/bluesky-is-down-today.RSS for NodeFri, 01 May 2026 05:25:37 GMTThu, 16 Apr 2026 18:34:30 GMT60<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:24:04 GMT]]>@mcc the worst are distributed p2p attacks like i watch in ipfs.

]]>https://board.circlewithadot.net/post/https://social.manalejandro.com/@ale@social.manalejandro.com/posts/567701750042044264/https://board.circlewithadot.net/post/https://social.manalejandro.com/@ale@social.manalejandro.com/posts/567701750042044264/Thu, 16 Apr 2026 20:24:04 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:23:48 GMT]]>@aeris @mcc and yet, despite the 43 min latency you’re reporting, we’ve been having a perfectly synchronous conversation for the last 15 minutes.

]]>https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416251715910565https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416251715910565Thu, 16 Apr 2026 20:23:48 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:23:32 GMT]]>@mcc@mastodon.social @jeromechoo@masto.ai And my new instance (migrating from Mastodon to Misskey exactly for this reason) is ALREADY filled with 600 dangling requests. At this point it doesn't generate any noticable delay, but only because the overall death rate is low. If a huge instance goes down, it would not be the same at all.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hqyi4ngsq0094https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hqyi4ngsq0094Thu, 16 Apr 2026 20:23:32 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:22:06 GMT]]>@jeromechoo@masto.ai @mcc@mastodon.social No, it not running fine. I ALREADY reported 43 minutes latency to deliver ANY MESSAGE on Mastodon. This "bug" (in fact bad design) is known since ages.
https://github.com/mastodon/mastodon/issues/12445

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hp4dkngsq0093https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hp4dkngsq0093Thu, 16 Apr 2026 20:22:06 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:21:12 GMT]]>@mcc@mastodon.social @jeromechoo@masto.ai At the end any workers just have 7% of "luck" (3 out of 42) to hit a down request, consuming resource for nothing for 2-3s, having no more time to schedule alive server, with 13.000 pending request because starvation, with many many alive request in those 13.000. Perhaps the 13.000th will be a alive one, but it will be delivered in only 43 minutes in average.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hnykbngsq0092https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hnykbngsq0092Thu, 16 Apr 2026 20:21:12 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:21:05 GMT]]>@aeris @mcc like I have already said — every post you’ve made just now has failed to deliver to several instances. Your instance is running just fine is it not?

If mastodon.social goes down. It would add ONE more failed delivery to the queue of thousands your instance is already managing.

]]>https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416241059902094https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416241059902094Thu, 16 Apr 2026 20:21:05 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:20:29 GMT]]>@nasser @mcc Thanks indeed! This is a great explanation.

My own takeaway is that Bluesky is a lost cause in terms of decentralization, because its architecture is designed to resist that outcome.

]]>https://board.circlewithadot.net/post/https://chaos.social/users/slothrop/statuses/116416238663415267https://board.circlewithadot.net/post/https://chaos.social/users/slothrop/statuses/116416238663415267Thu, 16 Apr 2026 20:20:29 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:18:28 GMT]]>@mcc@mastodon.social @aeris@firefish.imirhil.fr if that's really the case, if anything, that's an implementation problem. Mail servers have dealt with this problem for ages. That's why they have queues and per server exponentially increasing retry intervals. Push is not inherently bad.

]]>https://board.circlewithadot.net/post/https://federation.network/notes/al5hkfupj44600ojhttps://board.circlewithadot.net/post/https://federation.network/notes/al5hkfupj44600ojThu, 16 Apr 2026 20:18:28 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:18:07 GMT]]>@mcc@mastodon.social @jeromechoo@masto.ai After a while, you have 43 minutes latency for EVERY DELIVERY, even alive server. I experience that on my own Mastodon instance…

Link Preview Image
]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hk0c9ngsq0091https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hk0c9ngsq0091Thu, 16 Apr 2026 20:18:07 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:17:38 GMT]]>@breizh As of this second, Blacksky has resolved the issue. I don't know how.

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416227484312180https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416227484312180Thu, 16 Apr 2026 20:17:38 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:16:48 GMT]]>Updates

- Over the last two hours the problem has gone from "I don't see my posts" to "I see my posts 1 hour after I make them" to "17 minutes" to "3 minutes" to "it's fixed". I interpret this as the relay firehose pointer, whatever relay is in use right now, gradually catching up.

- I need to stress the above thread is a mix of fact (ATProto federation is duplicative and often brittle) and conjecture (I can't know what relay is being used internally by Blacksky except if Rudy tells us).

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416224224800921https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416224224800921Thu, 16 Apr 2026 20:16:48 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:16:11 GMT]]>@mcc@mastodon.social @jeromechoo@masto.ai Each post is a dangling request which will consume 3s of CPU time and so 10× consumption of 300ms for alive server, and planned for reschedule. After a while, all workers are just stuck with full of 3s waiting process, with starvation for alive requests.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hhispngsq008zhttps://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hhispngsq008zThu, 16 Apr 2026 20:16:11 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:14:11 GMT]]>@jeromechoo@masto.ai @mcc@mastodon.social It affect deliver to masto.ai because EACH of my post generate a dangling request, hiting timeout. After a while, my worker consume more time to dangling request taking 2-3s (hiting timeout) than trying to send content to masto.ai.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hexvgngsq008yhttps://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5hexvgngsq008yThu, 16 Apr 2026 20:14:11 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:13:19 GMT]]>@mcc@mastodon.social From what I understand of the protocol, they could just stop using a relay at all, but then it would increase the traffic on all the PDS that were scrapped by the relay until then, since the AppView would have to connect to each of those instead of the relay.

And did switching to another relay solved the issue?

]]>https://board.circlewithadot.net/post/https://pleroma.breizh.pm/objects/d26f21d7-6db9-4779-b58a-5e77c361590bhttps://board.circlewithadot.net/post/https://pleroma.breizh.pm/objects/d26f21d7-6db9-4779-b58a-5e77c361590bThu, 16 Apr 2026 20:13:19 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:12:19 GMT]]>@aeris @mcc this post you made probably just failed to deliver to a few instances. One of them could be mastodon.social if it was down.

How does that affect delivery to masto.ai?

mastodon.social being down is no different to your server than any other server being down. One request each.

]]>https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416206591605231https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416206591605231Thu, 16 Apr 2026 20:12:19 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:10:44 GMT]]>@mcc When I initially raised my eyebrows at Bluesky's notion of "federation", I was told that anyone can run a relay on a small cheap computer, it's dead easy, etc.…

]]>https://board.circlewithadot.net/post/https://cosocial.ca/users/timbray/statuses/116416200311414600https://board.circlewithadot.net/post/https://cosocial.ca/users/timbray/statuses/116416200311414600Thu, 16 Apr 2026 20:10:44 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:07:31 GMT]]>@mcc@mastodon.social @jeromechoo@masto.ai The trouble scale not to the down instance size, but to the alive instance size. The more it is active with many content generated, the fastest the background job queue fill with dangling content.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5h6dkkngsq008xhttps://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5h6dkkngsq008xThu, 16 Apr 2026 20:07:31 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:06:07 GMT]]>@mcc@mastodon.social @jeromechoo@masto.ai So a huge instance sent dozen of post per second (many content generated, but delivered only one) to another huge instance, with one background job per content to deliver.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5h4kb9ngsq008whttps://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5h4kb9ngsq008wThu, 16 Apr 2026 20:06:07 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:04:20 GMT]]>@jeromechoo@masto.ai @mcc@mastodon.social Yes, I know that. Trouble is not one content send to many, but many content sent to one.
Each post of one instance is sent only once to mastodon.social, but EACH post.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5h2a0fngsq008vhttps://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5h2a0fngsq008vThu, 16 Apr 2026 20:04:20 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:03:30 GMT]]>@aeris @mcc failed deliveries happen all the time. mastodon.social is just another instance. Since Mastodon and misskey operate on shared inboxes, the failed deliveries won’t scale sender failures by recipient instance size.

https://seb.jambor.dev/posts/understanding-activitypub/#:~:text=To%20combat%20this,our%20followers%20internally.

]]>https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416171869190345https://board.circlewithadot.net/post/https://masto.ai/users/jeromechoo/statuses/116416171869190345Thu, 16 Apr 2026 20:03:30 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 20:01:29 GMT]]>@mcc@mastodon.social And it's a vector attack in theory. You can bootstrap thousands of instance, just subscribing to as many account as possible, and then just shutdown your instance.
Any content from subscribed account will generate a background job to your down instance, then hiting timeout each time.
You can just flood instance like that to continue to overflow queue with dangling content.

]]>https://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5gylw4ngsq008shttps://board.circlewithadot.net/post/https://firefish.imirhil.fr/notes/al5gylw4ngsq008sThu, 16 Apr 2026 20:01:29 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 19:58:45 GMT]]>@mcc @esm I think we'll need to wait for the analysis and blog posts that follow.

]]>https://board.circlewithadot.net/post/https://hachyderm.io/users/thisismissem/statuses/116416153225904246https://board.circlewithadot.net/post/https://hachyderm.io/users/thisismissem/statuses/116416153225904246Thu, 16 Apr 2026 19:58:45 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 19:58:44 GMT]]>@esm @thisismissem I mean it's certainly possible that I am simply misinterpreting Rudy's comments about relays!… but all we ever get from Rudy are these vague gnomic comments, so this is about the best I can do. I'd rather him be spending his time sysadminining and writing Rust than writing up incident reports for public consumption but it does mean trying to figure out wtf is happening to my feed as a blacksky user is constant detective work

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416153154122615https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416153154122615Thu, 16 Apr 2026 19:58:44 GMT<![CDATA[Reply to Bluesky is down today. on Thu, 16 Apr 2026 19:57:03 GMT]]>@esm @thisismissem That's interesting, but

1. If it's true, why would the DDOS differentially impact third-party PDSes on Blacksky while Blacksky PDS runs at normal speed?

2. Did atproto go down because of a DOS or because of some side-effect of an attempt to move over to it as the primary relay?

One possibility is the failures I saw were *because* we switched from bluesky to atproto.africa, causing a short netlag period while atproto.africa caught up to the present? I don't know?

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416146549961134https://board.circlewithadot.net/post/https://mastodon.social/users/mcc/statuses/116416146549961134Thu, 16 Apr 2026 19:57:03 GMT