Apparently, subsidised code automation tools ("coding agents") that make it 10x easier to generate code is a big part of why GitHub has been having issues
-
What I meant above is that many of us had idly wondered whether GitHub's unreliability was because of MS's drive to use LLMs for coding, leading to a drop in quality. Instead they seem to be seriously mishandling a demand increase driven by their own actions
And it only partially makes sense as an explanation
They've been through this before. When MS bought GitHub and made it free, demand exploded. In theory, they should have all the expertise and capability to handle another massive increase
Normally scaling issues exhibit themselves a bit differently (normally) than the outages and outright data losses people have been seeing
So, while I believe GitHub managerial types when they say that the proximate cause is an increase in demand, that's unlikely to be the ultimate cause of what users are seeing
It's hard to speculate what the ultimate cause is when dealing with a system and organisation this big, but I can think of a few potential ones (could all be wrong, though):
-
Normally scaling issues exhibit themselves a bit differently (normally) than the outages and outright data losses people have been seeing
So, while I believe GitHub managerial types when they say that the proximate cause is an increase in demand, that's unlikely to be the ultimate cause of what users are seeing
It's hard to speculate what the ultimate cause is when dealing with a system and organisation this big, but I can think of a few potential ones (could all be wrong, though):
- They've lost the staff with the expertise needed to scale this. Up shit creek without a paddle in a canoe staffed by people who didn't realise they needed a paddle
- They're more constrained than they're letting on. Capacity and engineering resources are reserved for "AI" so corners are being cut
- GitHub as a system has serious flaws that only come into effect at this new scale and fixing them is a much bigger task than they expected.
Could be all of these. Could be none.
-
What I meant above is that many of us had idly wondered whether GitHub's unreliability was because of MS's drive to use LLMs for coding, leading to a drop in quality. Instead they seem to be seriously mishandling a demand increase driven by their own actions
And it only partially makes sense as an explanation
They've been through this before. When MS bought GitHub and made it free, demand exploded. In theory, they should have all the expertise and capability to handle another massive increase
@baldur yeah but they laid off thousands, and have turned over coding to genAI instead of experts at tuning and scaling. These are choices that take a toll on performance and uptime.
-
@baldur yeah but they laid off thousands, and have turned over coding to genAI instead of experts at tuning and scaling. These are choices that take a toll on performance and uptime.
@cczona Yeah, that has to have an effect somewhere.
-
What I meant above is that many of us had idly wondered whether GitHub's unreliability was because of MS's drive to use LLMs for coding, leading to a drop in quality. Instead they seem to be seriously mishandling a demand increase driven by their own actions
And it only partially makes sense as an explanation
They've been through this before. When MS bought GitHub and made it free, demand exploded. In theory, they should have all the expertise and capability to handle another massive increase
@baldur That Azure migration is just… yikes.
-
Normally scaling issues exhibit themselves a bit differently (normally) than the outages and outright data losses people have been seeing
So, while I believe GitHub managerial types when they say that the proximate cause is an increase in demand, that's unlikely to be the ultimate cause of what users are seeing
It's hard to speculate what the ultimate cause is when dealing with a system and organisation this big, but I can think of a few potential ones (could all be wrong, though):
@baldur
100%...started getting into more accidents...
"Well, that's just because we're driving more"
Or, maybe it's revealing something about your driving?
-
@baldur GitHub dropped their Actions pricing this year to get people off 3rd party runners, so it's weird for them to complain about foot pain after they pulled the trigger
@sanityinc @baldur they dropped actions pricing right after they tried to start charging people to run actions on their own hardware. Whining that they had 'real costs' from running the control plane.
-
What I meant above is that many of us had idly wondered whether GitHub's unreliability was because of MS's drive to use LLMs for coding, leading to a drop in quality. Instead they seem to be seriously mishandling a demand increase driven by their own actions
And it only partially makes sense as an explanation
They've been through this before. When MS bought GitHub and made it free, demand exploded. In theory, they should have all the expertise and capability to handle another massive increase
@baldur@toot.cafe Maybe I'm out of my mind, but looking at this graph, there appears to be a treatment and a subsequent effect: https://damrnelson.github.io/github-historical-uptime/
...which may or may not have to do with an internal culture that does things like this: https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion
and also this: https://www.propublica.org/article/microsoft-digital-escorts-pentagon-defense-department-china-hackers
-
@baldur@toot.cafe Maybe I'm out of my mind, but looking at this graph, there appears to be a treatment and a subsequent effect: https://damrnelson.github.io/github-historical-uptime/
...which may or may not have to do with an internal culture that does things like this: https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion
and also this: https://www.propublica.org/article/microsoft-digital-escorts-pentagon-defense-department-china-hackers@abucci I mean, yeah. The simplest explanation would be that they never actually managed to scale the thing properly while at the same time trying to migrate it to Azure and push a bunch of bullshit features and now it's just all coming to a head.
-
What I meant above is that many of us had idly wondered whether GitHub's unreliability was because of MS's drive to use LLMs for coding, leading to a drop in quality. Instead they seem to be seriously mishandling a demand increase driven by their own actions
And it only partially makes sense as an explanation
They've been through this before. When MS bought GitHub and made it free, demand exploded. In theory, they should have all the expertise and capability to handle another massive increase
@baldur if it was only because of LLM load, that graph of reliability wouldn't have immediately started when MS bought them
-
@baldur if it was only because of LLM load, that graph of reliability wouldn't have immediately started when MS bought them
@RangerRick Yeah, that's a good point.
-
- They've lost the staff with the expertise needed to scale this. Up shit creek without a paddle in a canoe staffed by people who didn't realise they needed a paddle
- They're more constrained than they're letting on. Capacity and engineering resources are reserved for "AI" so corners are being cut
- GitHub as a system has serious flaws that only come into effect at this new scale and fixing them is a much bigger task than they expected.
Could be all of these. Could be none.
@baldur There is a feel to it all. Scaling requires people who monitor, understand, test a system, find the bottlenecks, the creaky parts. It feels like all the plastic bits of GitHub are falling off.
-
R relay@relay.mycrowd.ca shared this topic
