Bluesky is down today.
-
And it's extremely relatable why Rudy took this shortcut of "build out our own stuff, but rely on Bluesky's components until we're forced to drop it": *Because standing up your own Bluesky stack is nightmarish!* It is a borderline miracle that a team his size made this work at all; I'm not sure a third team could replicate to the extent Blacksky has (and even on non-outage days, there are still large technical problems with Blacksky which cannot conveniently be fit in this thread).
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
-
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
@mcc I didn't know Hobbits were peer to peer
-
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
@mcc this is a really good breakdown, thank you for this thread
-
P2P is a world where naturally the more people use it, the faster and more resilient the network becomes. Load gets distributed. Working nodes talk to each other and ignore nonworking nodes. That's how the primitive, BitTorrent era systems worked.
Bluesky somehow applied superfancy alien future technology to invent P2P traffic jams. When one node goes down, the others go down because they depended on it. Because it's a mesh of interoperating microservices by different providers, not federation.
@mcc Capitalists always need a bottleneck so they can erect a tollbooth.
-
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
This is why I believe Bluesky was never meant to be federated. To create a Bluesky "instance", like Blacksky is heroically attempting, you have to perfectly duplicate every server Bluesky runs. But Bluesky is a business operating at a loss by burning unlimited-for-now VC cash. That has always implied only a business with unlimited VC cash can create an instance. Blacksky is succeeding. Except on days where they aren't.
-
@mcc this is a really good breakdown, thank you for this thread
I hope blacksky succeeds and gets their own complete version stood up, more options are better.
But it looks like there is a yardful of rakes for them to get through still.
-
This is why I believe Bluesky was never meant to be federated. To create a Bluesky "instance", like Blacksky is heroically attempting, you have to perfectly duplicate every server Bluesky runs. But Bluesky is a business operating at a loss by burning unlimited-for-now VC cash. That has always implied only a business with unlimited VC cash can create an instance. Blacksky is succeeding. Except on days where they aren't.
TLDR
1. My definition of "P2P" or "Federated" is that if server A goes down, servers B and C can still talk to each other.
2. Bluesky/"Atmosphere" fails at this because Blacksky (B) requires Bluesky (A) to talk to me (C).
3. In order for Blacksky to avert this, they have to do something unreasonable and expensive.
4. Blacksky someday *will* do this, but will depend heavily on massively overworking Rudy and a few other people. This may someday fail.
5. ActivityPub has problems, but not these
-
TLDR
1. My definition of "P2P" or "Federated" is that if server A goes down, servers B and C can still talk to each other.
2. Bluesky/"Atmosphere" fails at this because Blacksky (B) requires Bluesky (A) to talk to me (C).
3. In order for Blacksky to avert this, they have to do something unreasonable and expensive.
4. Blacksky someday *will* do this, but will depend heavily on massively overworking Rudy and a few other people. This may someday fail.
5. ActivityPub has problems, but not these
@mcc thank you for this whole thread, it really helps with understanding where BlueSky/ATProto/BlackSky/masto all fit into things, for me anyway
-
TLDR
1. My definition of "P2P" or "Federated" is that if server A goes down, servers B and C can still talk to each other.
2. Bluesky/"Atmosphere" fails at this because Blacksky (B) requires Bluesky (A) to talk to me (C).
3. In order for Blacksky to avert this, they have to do something unreasonable and expensive.
4. Blacksky someday *will* do this, but will depend heavily on massively overworking Rudy and a few other people. This may someday fail.
5. ActivityPub has problems, but not these
(And *how* does ActivityPub avert these problems? Well, ActivityPub has the "instance" abstraction. The federate-or-defederate relationships serve as a basic web of trust so some work, like moderation, doesn't have to be fully duplicated. Data is shared between instances only when a follow-relationship requires it, reducing work. Instances can still get too big and maintainers overworked, but you can fix that problem with more, smaller instances. As above, *there ARE no small Bluesky instances*)
-
There's a bit in Terry Jones' "Starship Titanic" where an alien gets caught in a traffic jam on Earth, and explodes "your transportation system is so poorly designed! when more people use it, it goes *slower*! you should design it so it goes *faster*— to accommodate the extra load!".
It's funny because the former seems like the obvious, natural way transportation works, and the latter seems to require magic alien future technology.
@mcc well you probably know that the transit system Utopia designed goes *way* slower than The Six-Hive Transport System, but with 100% less fatal traffic accidents
-
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
@mcc Very funny to me it took several tries to load this post so I could see it
-
@mcc Very funny to me it took several tries to load this post so I could see it
@misty try manually replacing bsky.app with blacksky.community. hm, i'll just do that in the post
-
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
@mcc love 2 have undisclosed dependencies in my federated independent network graph
-
@mcc Is Blacksky's stack the same code as the Bluesky one, or have they reimplemented it all from specs?
@darkling They are attempting to reimplement it all from specs, but that takes time, and Bluesky being fickle keeps forcing them to pause work on their Rust reimplementations and stand up git-cloned duplicates of Bluesky software instead, the work of maintaining which means they have less time to reimplement from spec
-
@mcc love 2 have undisclosed dependencies in my federated independent network graph
@mcc the rate at which we keep running into “oopsies what we said before wasn’t *technically* accurate, you see…” feels sus
-
TLDR
1. My definition of "P2P" or "Federated" is that if server A goes down, servers B and C can still talk to each other.
2. Bluesky/"Atmosphere" fails at this because Blacksky (B) requires Bluesky (A) to talk to me (C).
3. In order for Blacksky to avert this, they have to do something unreasonable and expensive.
4. Blacksky someday *will* do this, but will depend heavily on massively overworking Rudy and a few other people. This may someday fail.
5. ActivityPub has problems, but not these
@mcc@mastodon.social 2 is totally false.
I was able today to log in on BlackSky with my EuroSky PDS and keep talking with anybody currently not using BlueSky AppView, including people using BlueSky PDS but not BlueSky AppView. -
@mcc@mastodon.social 2 is totally false.
I was able today to log in on BlackSky with my EuroSky PDS and keep talking with anybody currently not using BlueSky AppView, including people using BlueSky PDS but not BlueSky AppView.@mcc@mastodon.social https://blacksky.community/profile/did:plc:z5wqufpi3akdylu2sqyzryqr/post/3mjmersqlu22a
This post was posted using BlackSky during BlueSky trouble, using a OAuth authentication from my EuroSky PDS, to somebody who create his account with EuroSky during the BlueSky shutdown. -
@mcc@mastodon.social https://blacksky.community/profile/did:plc:z5wqufpi3akdylu2sqyzryqr/post/3mjmersqlu22a
This post was posted using BlackSky during BlueSky trouble, using a OAuth authentication from my EuroSky PDS, to somebody who create his account with EuroSky during the BlueSky shutdown.@aeris I don't know what to tell you. It working on your machine does not cause it to work on my machine. In my own testing, right now, with the loop of "can blacksky view a post made using blacksky and hosted on a fully independent PDS", it takes about an hour for my own posts to show up in my own feed. Because Bluesky is made of many parts, it's possible the Bluesky relay was not having problems at the start of the Bluesky outage (when you did your test) but is now.
-
Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".
Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.
I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)
@mcc
I was having problems with it too. -
@mcc I’ve heard it described as “vertically decentralized” such that if any single layer goes down then the whole thing goes down
@mosssupply that's the terminology i like
