I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them.
-
@mitchellh Thank you for an interesting take! It got me thinking (from a statistics perspective) about MTBF and MTTR. The whole ”mean time” thing worries me. Time is strictly positive, so the median time must be higher than the mean time. Outliers will happen, and for MTBF that’s a good thing, while for MTTR it is not. In other words: a measurement of MTBF is pessimistic and conservative, while a measurement of MTTR is optimistic and a fluke away from disaster.
@mitchellh In hindsight I suppose no one is surprised at all, but do you have any memory of anyone taking that additional pass at basic reasoning at the time?
-
@totoroot @mitchellh Has this been a big problem for you in open source communities? LLM chatbots are a huge threat to open source, and as a community they tend to be more openly opposed to it.
Also FWIW, just hammering down literacy (if people know how these systems actually generate fluent explanation-shaped text) that goes a long way to mitigating adoption. Not full proof obviously looking at companies like OpenAI, but this does have a big impact.
@ambrosia_engine @totoroot @mitchellh Good point with the literacy. Even I am not in the industry I signed-up for two ML courses on my alma-mater at Masaryk University to better understand how it works. During courses I slowly lost any residual feeling of “magic” and “allmightyness” and stopped using LLM with exception for limited couching in coding which is my hobby (with a deliberately introduced friction and docs-checking). It helps when I can explain deeper insights to others.
-
@ambrosia_engine @totoroot @mitchellh Good point with the literacy. Even I am not in the industry I signed-up for two ML courses on my alma-mater at Masaryk University to better understand how it works. During courses I slowly lost any residual feeling of “magic” and “allmightyness” and stopped using LLM with exception for limited couching in coding which is my hobby (with a deliberately introduced friction and docs-checking). It helps when I can explain deeper insights to others.
@ambrosia_engine @totoroot @mitchellh + I consider very useful to go through algorithms like Word2Vec which is highly comprehensible and de-mystifying in some way. Even it is not the same LLMs do, the intuition of word embeddings is there very clear. It can reveal a lot of “the basic math only” that is actually going on in it. This is a great article with informative visuals: https://jalammar.github.io/illustrated-word2vec/
-
@ambrosia_engine @totoroot @mitchellh + I consider very useful to go through algorithms like Word2Vec which is highly comprehensible and de-mystifying in some way. Even it is not the same LLMs do, the intuition of word embeddings is there very clear. It can reveal a lot of “the basic math only” that is actually going on in it. This is a great article with informative visuals: https://jalammar.github.io/illustrated-word2vec/
@ambrosia_engine @totoroot @mitchellh And btw this is the whole (still minimized) GPT algorithm on one screen of python code by Karpathy: https://karpathy.ai/microgpt.html - stunning there is nearly everything that basic LLM needs (just without stolen data and optimizations, which actually made the crucial leap.)
Crazy that the world is upside
down because of something like this. -
@ambrosia_engine @totoroot @mitchellh + I consider very useful to go through algorithms like Word2Vec which is highly comprehensible and de-mystifying in some way. Even it is not the same LLMs do, the intuition of word embeddings is there very clear. It can reveal a lot of “the basic math only” that is actually going on in it. This is a great article with informative visuals: https://jalammar.github.io/illustrated-word2vec/
@tombarys @totoroot @mitchellh This is still part of the process, transformers still use embeddings. And it's a very good place to start.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh is always a challenge even if we dislike
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh Our company is very deep into it. And we have got people that argue like that. Ironically I work a lot with these systems as well. But I always perform a manual review of the code before creating the pull request. Also our pull requests are always reviewed by humans.
Especially during the first review, I often see stuff in the code that I really don't like, either from the way it is coded or from an efficiency standpoint or useless complexity.
And concerning the "full test coverage": Just Friday I fixed a bug that one of our human testers found. Before he tested it, I already had some gut feeling that there could be a problem in the code, so I told the tester about it. And my gut feeling was right. So after I fixed the bug, several automatically created tests failed ...
And concerning the tests: I also had situations where the LLM did some coding that was bad, so that already existing tests failed. And what did the LLM: It hadn't fixed the code but changed the test ...
At least in our department we always require human reviews, so I hope that the quality won't suffer.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh the plot to destroy software infrastructure that use #AI / #ThreeBodyProblem
-
@mitchellh One of the best descriptions I've heard lately was that it feels like "losing coworkers to dementia" as people adopt it, where everyone feels like they know everything, but when you talk with them in person or there is a problem that needs to be fixed _now_ it becomes very clear that the capability to do that has atrophied basically completely
@pojntfx @mitchellh
AI deskilling or covid brain damage; why don't we have both?
-
@pojntfx I lost a loved one to dementia, and have been saying for a while how similar LLM psychosis is to that experience.
Carina C. Zona (@cczona@hachyderm.io)
@Di4na @jenniferplusplus and there is already evidence emerging that engineers who depend on LLMs to write their code for them are eroding their skills. I would analogize it to early stage dementia. The person can't see how their judgement is gradually developing fissures that compromise their ability to function. Eventually it will become too clear to deny anymore. But right now they are increasingly impaired while no less confident in the comprehensiveness of their skills. It's the period when they present a big risk to self and others, because of the growing gap between reality and perception of competence. This person is letting LLMs draft most of their code, and fails to see that not continuing to hone their own skills as an active coder has personal consequences; and that doing so en masse poses societal consequences. What happens in a generation when there are virtually no engineers left who can review a LLM's outputs competently?
Hachyderm.io (hachyderm.io)
Carina C. Zona (@cczona@hachyderm.io)
@janl@narrativ.es I do feel in mourning. It's been very similar to when a relative developed dementia. The person I knew disappeared in jolting episodes, replaced by someone with their face but an unrecognizable personality and stripped of the values they'd held dear. It was deeply troubling and so, so sad to be around. In the past year I've been feeling that same sense of loss. Moving through the stages of grief for one person, then another, another. It's mentally exhausting. But when you care about someone or something, it's difficult to just decide to not let it matter.
Hachyderm.io (hachyderm.io)
They think they are fine, while the person you used to know is gradually replaced by someone unrecognisable. Professional de-skilling issues aside, a sadder dimension to me is how AI psychosis degrades not only cognition but also social skills. So relationships deteriorate too, and at scale that will ultimately become communities and solidarity deteriorating as well. It's been disturbing to see people I care about lose interest in connecting at a human level, while grasping for the empty flattery of a large language model. Is this our future, truly?
I wonder how long it will take for the rest of the world to a) realize what's going on, b) arrive at the objectively valid conclusion and c) act on it and see it through.
Recently past experience (COVID-19) shows that being a tall order.
Anyway, I wonder when, if so, people who actively rejected contact with LLMs will be in extremely high demand, being knowledge bearers in teaching and technical lead positions.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh worrying is for people who do not know the outcome. You know the outcome.
-
@mitchellh One of the best descriptions I've heard lately was that it feels like "losing coworkers to dementia" as people adopt it, where everyone feels like they know everything, but when you talk with them in person or there is a problem that needs to be fixed _now_ it becomes very clear that the capability to do that has atrophied basically completely
@pojntfx @mitchellh this nails how I have been feeling at work lately. As more people adopt and push AI, it's like seeing more people lose their abilities at a level similar to some senior relatives losing their faculties. Well worded.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
Conversely, there is entire cohorts of folks who are in depths of severe #Ai anxiety.
There is no point having a conversation with them because they are entirely irrational.
I worry.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh We've also learned the lesson (at least in security) to not mix code and data, yet all LLMs intrinsically repeat those mistakes.
We're living through a time of forgotten history, which is kind-of funny-but-not-really in computer science when the main crowd having learned these lessons is still around and - quite ironically - now often in middle to upper management.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh To a man with only a hammer every problem is a nail
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh let it burn. That's what they want that's what they get.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh code rot
-
@mitchellh One of the best descriptions I've heard lately was that it feels like "losing coworkers to dementia" as people adopt it, where everyone feels like they know everything, but when you talk with them in person or there is a problem that needs to be fixed _now_ it becomes very clear that the capability to do that has atrophied basically completely
@pojntfx @mitchellh I have seen the same behavior even with capable engineers that in the past I knew as strong designers and code quality champions now reduced to faith followers with 2 or 3 agents working on different branches of the same project at the same time and really not understanding or owning anything and still feeling top performers.
I do agree it looks very much like a dementia -
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh I thought devs get chewed out for prod going down, and now management is normalizing prod going down? I feel like I first observed the change when Elon bought Twitter, and everyone was applauding his cost cutting and that Twitter only went down a couple times, when in the past every single downtime was significant.
-
I strongly believe there are entire companies right now under heavy AI psychosis and its impossible to have rational conversations about it with them. I can't name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now its... the whole software development industry (maybe the whole world, really).
It's frightening, because the psychosis folks operate under an almost absolute "MTTR is all you need" mentality: "its fine to ship bugs because the agents will fix them so quickly and at a scale humans can't do!" We learned in infrastructure that MTTR is great but you can't yeet resilient systems entirely.
The main issue is I don't even know how to bring this up to people I know personally, because bringing this topic up leads to immediately dismissals like "no no, it has full test coverage" or "bug reports are going down" or something, which just don't paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happens so fast that nobody notices the underlying architecture decaying.
I worry.
@mitchellh obviously written by a pissed sentient AI agent.....
