I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh
In the 90s we only optimised to MTBF, MTTR wasn't considered terribly helpful. A software deployment could run for months without restarts or updates. But back then a person could hold an entire stack in their minds. I recall my first realisation of the total shift to MTTR with an ML based written record reader. We outsourced the ML, each instance could read only 1 record before dying. My boss was nonfussed, just restart it anew for every record... Infra is now just bubbles popping. -
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh@hachyderm.io Thanks for stating what many are seeing in our industry. Especially since your opinion bears a lot of weight.
I myself am trying to figure out how to voice concerns in the workplace and in the Open Source communities I participate in, while trying to not completely disregard the excitement people feel.
It becomes hard to argue with colleagues about the pros and cons of using LLMs in software engineering, when the CTO suggests an AI-first approach to anything, at any chance they get. -
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh I'm doing a PhD on the epistemic risks of LLMs. I used to work in industry and I see this too and it's so terrifying. Currently writing a paper looking at one of the mechanisms at play here, but how do we even talk to these people and get them help anymore? Especially when it's clear entire startups are affected.
-
@mitchellh @DukeDuke hadn't occurred to me that the AI psychosis may be a factor driving the enshittification of tech, but that makes perfect sense. I swear, COVID and genAI are our civilization's answer to Romans' lead pipes updated for the 21st century
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh I guess that's within the realm of what @pluralistic often refers to at "working well but failing badly"...
-
@mitchellh@hachyderm.io Thanks for stating what many are seeing in our industry. Especially since your opinion bears a lot of weight.
I myself am trying to figure out how to voice concerns in the workplace and in the Open Source communities I participate in, while trying to not completely disregard the excitement people feel.
It becomes hard to argue with colleagues about the pros and cons of using LLMs in software engineering, when the CTO suggests an AI-first approach to anything, at any chance they get.@totoroot @mitchellh Has this been a big problem for you in open source communities? LLM chatbots are a huge threat to open source, and as a community they tend to be more openly opposed to it.
Also FWIW, just hammering down literacy (if people know how these systems actually generate fluent explanation-shaped text) that goes a long way to mitigating adoption. Not full proof obviously looking at companies like OpenAI, but this does have a big impact.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh the other problem is that all of this depends on a very small number of capitalist giants with billions of debts and a broken business model. Everyone is acting like claude will fix bugs in spaghetti code forever, but all this could be in deep bankruptcy in a few years, with no one able to fix slop bugs anymore. That will be fun. So many popcorn.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh Thank you for an interesting take! It got me thinking (from a statistics perspective) about MTBF and MTTR. The whole ”mean time” thing worries me. Time is strictly positive, so the median time must be higher than the mean time. Outliers will happen, and for MTBF that’s a good thing, while for MTTR it is not. In other words: a measurement of MTBF is pessimistic and conservative, while a measurement of MTTR is optimistic and a fluke away from disaster.
-
@mitchellh Thank you for an interesting take! It got me thinking (from a statistics perspective) about MTBF and MTTR. The whole ”mean time” thing worries me. Time is strictly positive, so the median time must be higher than the mean time. Outliers will happen, and for MTBF that’s a good thing, while for MTTR it is not. In other words: a measurement of MTBF is pessimistic and conservative, while a measurement of MTTR is optimistic and a fluke away from disaster.
@mitchellh In hindsight I suppose no one is surprised at all, but do you have any memory of anyone taking that additional pass at basic reasoning at the time?
-
@totoroot @mitchellh Has this been a big problem for you in open source communities? LLM chatbots are a huge threat to open source, and as a community they tend to be more openly opposed to it.
Also FWIW, just hammering down literacy (if people know how these systems actually generate fluent explanation-shaped text) that goes a long way to mitigating adoption. Not full proof obviously looking at companies like OpenAI, but this does have a big impact.
@ambrosia_engine @totoroot @mitchellh Good point with the literacy. Even I am not in the industry I signed-up for two ML courses on my alma-mater at Masaryk University to better understand how it works. During courses I slowly lost any residual feeling of “magic” and “allmightyness” and stopped using LLM with exception for limited couching in coding which is my hobby (with a deliberately introduced friction and docs-checking). It helps when I can explain deeper insights to others.
-
@ambrosia_engine @totoroot @mitchellh Good point with the literacy. Even I am not in the industry I signed-up for two ML courses on my alma-mater at Masaryk University to better understand how it works. During courses I slowly lost any residual feeling of “magic” and “allmightyness” and stopped using LLM with exception for limited couching in coding which is my hobby (with a deliberately introduced friction and docs-checking). It helps when I can explain deeper insights to others.
@ambrosia_engine @totoroot @mitchellh + I consider very useful to go through algorithms like Word2Vec which is highly comprehensible and de-mystifying in some way. Even it is not the same LLMs do, the intuition of word embeddings is there very clear. It can reveal a lot of “the basic math only” that is actually going on in it. This is a great article with informative visuals: https://jalammar.github.io/illustrated-word2vec/
-
@ambrosia_engine @totoroot @mitchellh + I consider very useful to go through algorithms like Word2Vec which is highly comprehensible and de-mystifying in some way. Even it is not the same LLMs do, the intuition of word embeddings is there very clear. It can reveal a lot of “the basic math only” that is actually going on in it. This is a great article with informative visuals: https://jalammar.github.io/illustrated-word2vec/
@ambrosia_engine @totoroot @mitchellh And btw this is the whole (still minimized) GPT algorithm on one screen of python code by Karpathy: https://karpathy.ai/microgpt.html - stunning there is nearly everything that basic LLM needs (just without stolen data and optimizations, which actually made the crucial leap.)
Crazy that the world is upside
down because of something like this. -
@ambrosia_engine @totoroot @mitchellh + I consider very useful to go through algorithms like Word2Vec which is highly comprehensible and de-mystifying in some way. Even it is not the same LLMs do, the intuition of word embeddings is there very clear. It can reveal a lot of “the basic math only” that is actually going on in it. This is a great article with informative visuals: https://jalammar.github.io/illustrated-word2vec/
@tombarys @totoroot @mitchellh This is still part of the process, transformers still use embeddings. And it's a very good place to start.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh is always a challenge even if we dislike
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh Our company is very deep into it. And we have got people that argue like that. Ironically I work a lot with these systems as well. But I always perform a manual review of the code before creating the pull request. Also our pull requests are always reviewed by humans.
Especially during the first review, I often see stuff in the code that I really don't like, either from the way it is coded or from an efficiency standpoint or useless complexity.
And concerning the "full test coverage": Just Friday I fixed a bug that one of our human testers found. Before he tested it, I already had some gut feeling that there could be a problem in the code, so I told the tester about it. And my gut feeling was right. So after I fixed the bug, several automatically created tests failed ...
And concerning the tests: I also had situations where the LLM did some coding that was bad, so that already existing tests failed. And what did the LLM: It hadn't fixed the code but changed the test ...
At least in our department we always require human reviews, so I hope that the quality won't suffer.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh the plot to destroy software infrastructure that use #AI / #ThreeBodyProblem
-
@mitchellh One of the best descriptions I've heard lately was that it feels like "losing coworkers to dementia" as people adopt it, where everyone feels like they know everything, but when you talk with them in person or there is a problem that needs to be fixed _now_ it becomes very clear that the capability to do that has atrophied basically completely
@pojntfx @mitchellh
AI deskilling or covid brain damage; why don't we have both?
-
@pojntfx I lost a loved one to dementia, and have been saying for a while how similar LLM psychosis is to that experience.
Carina C. Zona (@cczona@hachyderm.io)
@Di4na @jenniferplusplus and there is already evidence emerging that engineers who depend on LLMs to write their code for them are eroding their skills. I would analogize it to early stage dementia. The person can't see how their judgement is gradually developing fissures that compromise their ability to function. Eventually it will become too clear to deny anymore. But right now they are increasingly impaired while no less confident in the comprehensiveness of their skills. It's the period when they present a big risk to self and others, because of the growing gap between reality and perception of competence. This person is letting LLMs draft most of their code, and fails to see that not continuing to hone their own skills as an active coder has personal consequences; and that doing so en masse poses societal consequences. What happens in a generation when there are virtually no engineers left who can review a LLM's outputs competently?
Hachyderm.io (hachyderm.io)
Carina C. Zona (@cczona@hachyderm.io)
@janl@narrativ.es I do feel in mourning. It's been very similar to when a relative developed dementia. The person I knew disappeared in jolting episodes, replaced by someone with their face but an unrecognizable personality and stripped of the values they'd held dear. It was deeply troubling and so, so sad to be around. In the past year I've been feeling that same sense of loss. Moving through the stages of grief for one person, then another, another. It's mentally exhausting. But when you care about someone or something, it's difficult to just decide to not let it matter.
Hachyderm.io (hachyderm.io)
They think they are fine, while the person you used to know is gradually replaced by someone unrecognisable. Professional de-skilling issues aside, a sadder dimension to me is how AI psychosis degrades not only cognition but also social skills. So relationships deteriorate too, and at scale that will ultimately become communities and solidarity deteriorating as well. It's been disturbing to see people I care about lose interest in connecting at a human level, while grasping for the empty flattery of a large language model. Is this our future, truly?
I wonder how long it will take for the rest of the world to a) realize what's going on, b) arrive at the objectively valid conclusion and c) act on it and see it through.
Recently past experience (COVID-19) shows that being a tall order.
Anyway, I wonder when, if so, people who actively rejected contact with LLMs will be in extremely high demand, being knowledge bearers in teaching and technical lead positions.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh worrying is for people who do not know the outcome. You know the outcome.
-
@mitchellh One of the best descriptions I've heard lately was that it feels like "losing coworkers to dementia" as people adopt it, where everyone feels like they know everything, but when you talk with them in person or there is a problem that needs to be fixed _now_ it becomes very clear that the capability to do that has atrophied basically completely
@pojntfx @mitchellh this nails how I have been feeling at work lately. As more people adopt and push AI, it's like seeing more people lose their abilities at a level similar to some senior relatives losing their faculties. Well worded.
