Bluesky is down today.
-
@miss_rodent Blacksky claims to have evaded that trap by duplicating literally every level of the Bluesky stack.
@mcc Oh okay, that's neat, didn't know that.
... does raise some eyebrows about thm also being down then, though, yeah. -
Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".
Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.
I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)
@mcc yeah, looks like blacksky got sideswiped or whatever
I guess there's only so much Rudy and his pirate crew can do.
I'm expecting blacksky will build back better
-
Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".
Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.
I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)
There's a bit in Terry Jones' "Starship Titanic" where an alien gets caught in a traffic jam on Earth, and explodes "your transportation system is so poorly designed! when more people use it, it goes *slower*! you should design it so it goes *faster*— to accommodate the extra load!".
It's funny because the former seems like the obvious, natural way transportation works, and the latter seems to require magic alien future technology.
-
There's a bit in Terry Jones' "Starship Titanic" where an alien gets caught in a traffic jam on Earth, and explodes "your transportation system is so poorly designed! when more people use it, it goes *slower*! you should design it so it goes *faster*— to accommodate the extra load!".
It's funny because the former seems like the obvious, natural way transportation works, and the latter seems to require magic alien future technology.
P2P is a world where naturally the more people use it, the faster and more resilient the network becomes. Load gets distributed. Working nodes talk to each other and ignore nonworking nodes. That's how the primitive, BitTorrent era systems worked.
Bluesky somehow applied superfancy alien future technology to invent P2P traffic jams. When one node goes down, the others go down because they depended on it. Because it's a mesh of interoperating microservices by different providers, not federation.
-
P2P is a world where naturally the more people use it, the faster and more resilient the network becomes. Load gets distributed. Working nodes talk to each other and ignore nonworking nodes. That's how the primitive, BitTorrent era systems worked.
Bluesky somehow applied superfancy alien future technology to invent P2P traffic jams. When one node goes down, the others go down because they depended on it. Because it's a mesh of interoperating microservices by different providers, not federation.
@mcc sounds like it's composing with a big "and" when it should be composing with a big "or"
-
P2P is a world where naturally the more people use it, the faster and more resilient the network becomes. Load gets distributed. Working nodes talk to each other and ignore nonworking nodes. That's how the primitive, BitTorrent era systems worked.
Bluesky somehow applied superfancy alien future technology to invent P2P traffic jams. When one node goes down, the others go down because they depended on it. Because it's a mesh of interoperating microservices by different providers, not federation.
@mcc I’ve heard it described as “vertically decentralized” such that if any single layer goes down then the whole thing goes down
-
P2P is a world where naturally the more people use it, the faster and more resilient the network becomes. Load gets distributed. Working nodes talk to each other and ignore nonworking nodes. That's how the primitive, BitTorrent era systems worked.
Bluesky somehow applied superfancy alien future technology to invent P2P traffic jams. When one node goes down, the others go down because they depended on it. Because it's a mesh of interoperating microservices by different providers, not federation.
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
-
Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".
Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.
I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)
@mcc yeah blacksky was down for me today, very on and off
-
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
@mcc Is Blacksky's stack the same code as the Bluesky one, or have they reimplemented it all from specs?
-
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
@mcc hilariously this opened in my Bsky client and the outage is glorious
-
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
Now, interestingly, this means that Blacksky users can continue talking to Blacksky users. I can read Rudy's posts on Blacksky. Because that bypasses the relay. But¹ to read my *own* posts, *on a self-hosted PDS*, Bluesky is apparently required, because Blacksky relies on Bluesky's "relay" to scrape my PDS before it gets added to the Blacksky appview database.
¹ (if I'm interpreting Rudy's posts correctly, hardly a guarantee)
-
Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".
Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.
I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)
@mcc for me, blacksky.community seems to query
api.bsky.appfor posts, which would suggest they aren't using their own AppView..? -
@mcc for me, blacksky.community seems to query
api.bsky.appfor posts, which would suggest they aren't using their own AppView..?@ptrc "lol"
All I know is what Blacksky claims.
-
Now, interestingly, this means that Blacksky users can continue talking to Blacksky users. I can read Rudy's posts on Blacksky. Because that bypasses the relay. But¹ to read my *own* posts, *on a self-hosted PDS*, Bluesky is apparently required, because Blacksky relies on Bluesky's "relay" to scrape my PDS before it gets added to the Blacksky appview database.
¹ (if I'm interpreting Rudy's posts correctly, hardly a guarantee)
@mcc I don't use bsky and haven't looked at the tech for awhile, but isn't there a thing about federating relays? I thought relays were fairly lightweight.
-
Now, interestingly, this means that Blacksky users can continue talking to Blacksky users. I can read Rudy's posts on Blacksky. Because that bypasses the relay. But¹ to read my *own* posts, *on a self-hosted PDS*, Bluesky is apparently required, because Blacksky relies on Bluesky's "relay" to scrape my PDS before it gets added to the Blacksky appview database.
¹ (if I'm interpreting Rudy's posts correctly, hardly a guarantee)
And it's extremely relatable why Rudy took this shortcut of "build out our own stuff, but rely on Bluesky's components until we're forced to drop it": *Because standing up your own Bluesky stack is nightmarish!* It is a borderline miracle that a team his size made this work at all; I'm not sure a third team could replicate to the extent Blacksky has (and even on non-outage days, there are still large technical problems with Blacksky which cannot conveniently be fit in this thread).
-
@mcc I don't use bsky and haven't looked at the tech for awhile, but isn't there a thing about federating relays? I thought relays were fairly lightweight.
@zrail "federate" in bluesky world means "duplicate completely"
-
@zrail "federate" in bluesky world means "duplicate completely"
@mcc cool cool cool this is fine
-
And it's extremely relatable why Rudy took this shortcut of "build out our own stuff, but rely on Bluesky's components until we're forced to drop it": *Because standing up your own Bluesky stack is nightmarish!* It is a borderline miracle that a team his size made this work at all; I'm not sure a third team could replicate to the extent Blacksky has (and even on non-outage days, there are still large technical problems with Blacksky which cannot conveniently be fit in this thread).
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
-
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
@mcc I didn't know Hobbits were peer to peer
-
Because this is the other "we used future alien technology to make it worse" thing about Bluesky.
In the "natural", Hobbesian form of P2P, the more nodes you add the less work per node you need to do, because of work sharing.
But Bluesky's "federation" is like blockchain. When you create a second "instance", that instance must duplicate *literally all the work* of the first instance. It must scrape all the posts itself. It must archive all the posts itself. It must CSAM-scan the posts itself.
@mcc this is a really good breakdown, thank you for this thread
