Bluesky is down today.
-
This appears to be the explanation:
Rudolph Fraser. (@rude1.blacksky.team)
Even their relay seems down(?) Trying to switch some things to use atproto.africa https://atproto.africa
Blacksky (blacksky.community)
In Bluesky, the PDS talks to the relay talks to the appview goes to the client. Blacksky set up all four last year. But they only deployed their PDS and client, at first. They used Bluesky's relay and appview. This wasn't clearly disclosed. Then there was a censorship scare, and they switched to their own appview. But apparently they're still using Bluesky's relay. This wasn't clearly disclosed. Now relay death kills Blacksky.
@mcc love 2 have undisclosed dependencies in my federated independent network graph
-
@mcc Is Blacksky's stack the same code as the Bluesky one, or have they reimplemented it all from specs?
@darkling They are attempting to reimplement it all from specs, but that takes time, and Bluesky being fickle keeps forcing them to pause work on their Rust reimplementations and stand up git-cloned duplicates of Bluesky software instead, the work of maintaining which means they have less time to reimplement from spec
-
@mcc love 2 have undisclosed dependencies in my federated independent network graph
@mcc the rate at which we keep running into “oopsies what we said before wasn’t *technically* accurate, you see…” feels sus
-
TLDR
1. My definition of "P2P" or "Federated" is that if server A goes down, servers B and C can still talk to each other.
2. Bluesky/"Atmosphere" fails at this because Blacksky (B) requires Bluesky (A) to talk to me (C).
3. In order for Blacksky to avert this, they have to do something unreasonable and expensive.
4. Blacksky someday *will* do this, but will depend heavily on massively overworking Rudy and a few other people. This may someday fail.
5. ActivityPub has problems, but not these
@mcc@mastodon.social 2 is totally false.
I was able today to log in on BlackSky with my EuroSky PDS and keep talking with anybody currently not using BlueSky AppView, including people using BlueSky PDS but not BlueSky AppView. -
@mcc@mastodon.social 2 is totally false.
I was able today to log in on BlackSky with my EuroSky PDS and keep talking with anybody currently not using BlueSky AppView, including people using BlueSky PDS but not BlueSky AppView.@mcc@mastodon.social https://blacksky.community/profile/did:plc:z5wqufpi3akdylu2sqyzryqr/post/3mjmersqlu22a
This post was posted using BlackSky during BlueSky trouble, using a OAuth authentication from my EuroSky PDS, to somebody who create his account with EuroSky during the BlueSky shutdown. -
@mcc@mastodon.social https://blacksky.community/profile/did:plc:z5wqufpi3akdylu2sqyzryqr/post/3mjmersqlu22a
This post was posted using BlackSky during BlueSky trouble, using a OAuth authentication from my EuroSky PDS, to somebody who create his account with EuroSky during the BlueSky shutdown.@aeris I don't know what to tell you. It working on your machine does not cause it to work on my machine. In my own testing, right now, with the loop of "can blacksky view a post made using blacksky and hosted on a fully independent PDS", it takes about an hour for my own posts to show up in my own feed. Because Bluesky is made of many parts, it's possible the Bluesky relay was not having problems at the start of the Bluesky outage (when you did your test) but is now.
-
Bluesky is down today. "Hah", I think, "since I use a self-hosted PDS for posting and Blacksky for viewing posts, I can go on using the service just fine".
Blacksky can't show me my own posts. I can make them and they show up on my PDS, but my profile shows none more recent than last night.
I wonder if Blacksky is coincidentally having server problems, or if Blacksky has a still-undisclosed dependency on Bluesky services. (It *does* have disclosed use of Bluesky moderation; maybe that's it.)
@mcc
I was having problems with it too. -
@mcc I’ve heard it described as “vertically decentralized” such that if any single layer goes down then the whole thing goes down
@mosssupply that's the terminology i like
-
@aeris I don't know what to tell you. It working on your machine does not cause it to work on my machine. In my own testing, right now, with the loop of "can blacksky view a post made using blacksky and hosted on a fully independent PDS", it takes about an hour for my own posts to show up in my own feed. Because Bluesky is made of many parts, it's possible the Bluesky relay was not having problems at the start of the Bluesky outage (when you did your test) but is now.
@mcc@mastodon.social It would probably be the same here on the Fediverse. I already report to Mastodon they have huge problem with down instance, and if a huge one stop responding, I expect many delay all over other instances, because down instance will fill all background job with dangling request
https://github.com/mastodon/mastodon/issues/12445 -
@mcc@mastodon.social It would probably be the same here on the Fediverse. I already report to Mastodon they have huge problem with down instance, and if a huge one stop responding, I expect many delay all over other instances, because down instance will fill all background job with dangling request
https://github.com/mastodon/mastodon/issues/12445@mcc@mastodon.social Even on fully (technically) distributed system, a huge sub-system will generate trouble everywhere in case of trouble. See how when Amazon or Cloudflare on trouble they can impact other sub-system, even if Internet is fully distributed. Such generated trouble is from concentrated node more than not decentralized protocol.
-
(And *how* does ActivityPub avert these problems? Well, ActivityPub has the "instance" abstraction. The federate-or-defederate relationships serve as a basic web of trust so some work, like moderation, doesn't have to be fully duplicated. Data is shared between instances only when a follow-relationship requires it, reducing work. Instances can still get too big and maintainers overworked, but you can fix that problem with more, smaller instances. As above, *there ARE no small Bluesky instances*)
@mcc fwiw, blacksky have had a relay for ages, they were one of the first to set one up, so I'm not sure what lead to this situation: https://atproto.africa/
Full list of services: https://docs.blacksky.community/list-of-our-services
-
@mcc@mastodon.social Even on fully (technically) distributed system, a huge sub-system will generate trouble everywhere in case of trouble. See how when Amazon or Cloudflare on trouble they can impact other sub-system, even if Internet is fully distributed. Such generated trouble is from concentrated node more than not decentralized protocol.
@aeris You have just put your finger on the difference between decentralized and distributed!!! Amazon is distributed but not decentralized. Bluesky is distributed but not decentralized. Fedi is both.
-
This is why I believe Bluesky was never meant to be federated. To create a Bluesky "instance", like Blacksky is heroically attempting, you have to perfectly duplicate every server Bluesky runs. But Bluesky is a business operating at a loss by burning unlimited-for-now VC cash. That has always implied only a business with unlimited VC cash can create an instance. Blacksky is succeeding. Except on days where they aren't.
@mcc I want some of that unlimited VC cash):
-
@aeris You have just put your finger on the difference between decentralized and distributed!!! Amazon is distributed but not decentralized. Bluesky is distributed but not decentralized. Fedi is both.
@mcc@mastodon.social Fedi is not both. And worse, by design, Fediverse is LESS distributed, because a down instance will stuck ALL other instance with dangling job, because it's a push protocol and not a pull one.
-
@mcc fwiw, blacksky have had a relay for ages, they were one of the first to set one up, so I'm not sure what lead to this situation: https://atproto.africa/
Full list of services: https://docs.blacksky.community/list-of-our-services
@thisismissem Yes, they have had a relay for over a year. But they're apparently not using it, similar to how they had an appview for about six months before they started using it. My source for this is Rudy saying, in the link upthread, that current blacksky service outages are due to the Bluesky relay going down and he is trying to "switch over to atproto.africa for some things". Implying he was not using atproto.africa for those things 24 hours ago.
-
@eestileib @mcc I'm no expert but it honestly sounds like a terrible way to build a network. or at least a pretty confounding way to build a network that you intend to be federated and decentralized in any capacity...
-
(And *how* does ActivityPub avert these problems? Well, ActivityPub has the "instance" abstraction. The federate-or-defederate relationships serve as a basic web of trust so some work, like moderation, doesn't have to be fully duplicated. Data is shared between instances only when a follow-relationship requires it, reducing work. Instances can still get too big and maintainers overworked, but you can fix that problem with more, smaller instances. As above, *there ARE no small Bluesky instances*)
@mcc Not an AT protocol expert, as I always thought Actvity Pub was a better approach, but I also was always skeptical of the house that Jack built. I felt it was a bit of grift to get people feeling they we were distributed while still being survieled by the mother-ship. The crypto linked VC funding didn't do much to ease those fears. However I don't cheer lead downtime either because I know how much that sucks. Hope you're up soon.
-
@mcc@mastodon.social Fedi is not both. And worse, by design, Fediverse is LESS distributed, because a down instance will stuck ALL other instance with dangling job, because it's a push protocol and not a pull one.
@mcc@mastodon.social Each message from my instance will generate a job to mastodon.social, because at least one ppl there follow me. If mastodon.social vanish, my own instance will start to hang for ALL message I post.
-
@mcc@mastodon.social Fedi is not both. And worse, by design, Fediverse is LESS distributed, because a down instance will stuck ALL other instance with dangling job, because it's a push protocol and not a pull one.
@aeris I don't think you know what you're talking about, and if mastodon.social goes down firefish.imirhill.fr can continue to talk to digipres.club.
-
@thisismissem Yes, they have had a relay for over a year. But they're apparently not using it, similar to how they had an appview for about six months before they started using it. My source for this is Rudy saying, in the link upthread, that current blacksky service outages are due to the Bluesky relay going down and he is trying to "switch over to atproto.africa for some things". Implying he was not using atproto.africa for those things 24 hours ago.
@mcc oh! Interesting, I'd just assumed they were already using it.
