I wish I could recommend this piece more, because it makes a bunch of great points, but the "normal technology" case feels misleading to me.
-
Could be sample bias, of course. I only loosely follow the science, and my audience obviously leans heavily skeptical at this point. I wouldn't pretend to *know* that the most dire predictions will come true. I'd much, much rather be conclusively proven wrong about this.
But I'm still waiting.
@glyph this thread needs to be an essay, and then a research hypothesis.
I very much feel like I’m watching the last 35 years of my ever-enshittifying social network exposure, sped up 10x and replayed.
In 1991 I remember having the flash of insight - without the life experience to really go into it deeply then - that the way nascent social network tech constrained and shaped interaction was going to force a mass cognitive adaptation for which we were not ready.
-
@glyph this thread needs to be an essay, and then a research hypothesis.
I very much feel like I’m watching the last 35 years of my ever-enshittifying social network exposure, sped up 10x and replayed.
In 1991 I remember having the flash of insight - without the life experience to really go into it deeply then - that the way nascent social network tech constrained and shaped interaction was going to force a mass cognitive adaptation for which we were not ready.
In 2021, we were still suffering the consequences of that, and still not sufficiently adapted to have avoided whatever the fuck is now driving our geopolitical dystopia engine.
And then suddenly our devolved capacity for social cognition had to deal with the fact that dealing with any humans at any distance far enough away that you couldn’t *lick* them came with no assurance that there even was a human there.
-
@delta_vee @kirakira @glyph Leaded gasoline.
@bluewinds @delta_vee @kirakira @glyph I don't think the analogies are good because asbestos is a fantastic insulator, lead is a really helpful additive for petrol and makes fantastic pigments and is really convenient for piping... and the hidden side-effects are the problem. Whereas LLMs _don't_ deliver that primary benefit
LLMs are more like... cheap laminate flooring, produced with wood pulp harvested unsustainably from old-growth forests and made by grossly exploited factory workers overseas... superficially convenient when remodelling your kitchen and rapidly ubiquitous but also quite unsatisfying and a right faff to work around once it's established
-
@bluewinds @delta_vee @kirakira @glyph I don't think the analogies are good because asbestos is a fantastic insulator, lead is a really helpful additive for petrol and makes fantastic pigments and is really convenient for piping... and the hidden side-effects are the problem. Whereas LLMs _don't_ deliver that primary benefit
LLMs are more like... cheap laminate flooring, produced with wood pulp harvested unsustainably from old-growth forests and made by grossly exploited factory workers overseas... superficially convenient when remodelling your kitchen and rapidly ubiquitous but also quite unsatisfying and a right faff to work around once it's established
@bluewinds @delta_vee @kirakira @glyph this post is brought to you by our kitchen floor
-
@bluewinds @delta_vee @kirakira @glyph I don't think the analogies are good because asbestos is a fantastic insulator, lead is a really helpful additive for petrol and makes fantastic pigments and is really convenient for piping... and the hidden side-effects are the problem. Whereas LLMs _don't_ deliver that primary benefit
LLMs are more like... cheap laminate flooring, produced with wood pulp harvested unsustainably from old-growth forests and made by grossly exploited factory workers overseas... superficially convenient when remodelling your kitchen and rapidly ubiquitous but also quite unsatisfying and a right faff to work around once it's established
@jackeric @bluewinds @kirakira @glyph Cheap laminate floors aren't a cognitohazard though (unless you're in interior design
-
@bluewinds @delta_vee @kirakira @glyph this post is brought to you by our kitchen floor
@jackeric @bluewinds @delta_vee @kirakira heh. I am not sure I 100% agree with your framing but all the analogies fall short (after all I do not think we have GOOD evidence that LLMs do any of these things, just hints) and this is an interesting contribution to the pile. but I definitely was thinking "wow it sounds like jack is thinking about laminate flooring really hard" the whole time I was reading it
-
If I could use another inaccurate metaphor, AI psychosis is the "instant decapitation" industrial accident with this new technology. And indeed, most people having industrial accidents are not instantly decapitated. But they might get a scrape, or lose a finger, or an eye. And an infected scrape can still kill you, but it won't look like the decapitation. It looks like you didn't take very good care of yourself. Didn't wash the cut. Didn't notice it fast enough. Skill issue.
@glyph
Here's an industrial accident that's easy to miss:A hydraulic fluid line bursts while you're working on a machine, injecting toxic and/or hot liquid under your skin at high pressure.
https://en.wikipedia.org/wiki/High_pressure_injection_injury
"Although the initial wound often seems minor, the unseen, internal damage can be severe. With hydraulic fluids, paint, and detergents, these injuries are extremely serious as most hydraulic fluids and organic solvents are highly toxic." -
1. YES THEY ARE.
They are vibe-coding mission-critical AWS modules. They are generating tech debt at scale. They don't THINK that that's what they're doing. Do you think most programmers conceive of their daily (non-LLM) activities as "putting in lots of bugs"? No, that is never what we say we're doing. Yet, we turn around, and there all the bugs are.
With LLMs, we can look at the mission-critical AWS modules and ask after the fact, were they vibe-coded? AWS says yes https://arstechnica.com/civis/threads/after-outages-amazon-to-make-senior-engineers-sign-off-on-ai-assisted-changes.1511983/
Having read over Doctorow's rant-du-jour twice now, I do think when he said "they" were not vibe coding mission-critial AWS modules", he was referring to the "they" in the previous paragraph, being developers he's spoken to, some of whom were friends he knows well.
So.... could be very differently skilled people from "some hack in a code assembly shop driving at a reckless pace because Amazon stock needs a bump".
It's all back to, though, defining "AI".
-
@glyph
Here's an industrial accident that's easy to miss:A hydraulic fluid line bursts while you're working on a machine, injecting toxic and/or hot liquid under your skin at high pressure.
https://en.wikipedia.org/wiki/High_pressure_injection_injury
"Although the initial wound often seems minor, the unseen, internal damage can be severe. With hydraulic fluids, paint, and detergents, these injuries are extremely serious as most hydraulic fluids and organic solvents are highly toxic."@dec23k okay definitely not clicking on that link, yeesh
-
Having read over Doctorow's rant-du-jour twice now, I do think when he said "they" were not vibe coding mission-critial AWS modules", he was referring to the "they" in the previous paragraph, being developers he's spoken to, some of whom were friends he knows well.
So.... could be very differently skilled people from "some hack in a code assembly shop driving at a reckless pace because Amazon stock needs a bump".
It's all back to, though, defining "AI".
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
-
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
@johannab So yes, maybe his contacts are transcendentally better programmers than mine, and they've ascended to a plane of subjective self-assessment beyond mere mortals, but if they're anything like the (extremely skilled, extremely experienced) people I've watched fall into this trap, I'm highly skeptical
-
@johannab So yes, maybe his contacts are transcendentally better programmers than mine, and they've ascended to a plane of subjective self-assessment beyond mere mortals, but if they're anything like the (extremely skilled, extremely experienced) people I've watched fall into this trap, I'm highly skeptical
@johannab the AWS link was to showcase that even AWS itself can't prevent vibe-coding their mission-critical modules, and presumably a few skilled practitioners work there.
-
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
@glyph Fair, for sure.
I just realized when reading it over that was a spot there could be a disconnect between the "they" being referred to in the essay narrative as written.
I feel like my immediate, 1-degree friends, acquaintances and colleagues include amongst them all the theoretical levels of self-awareness we could speak to, and indeed, *I* can't tell one from the other without more examination of context.
-
@glyph Fair, for sure.
I just realized when reading it over that was a spot there could be a disconnect between the "they" being referred to in the essay narrative as written.
I feel like my immediate, 1-degree friends, acquaintances and colleagues include amongst them all the theoretical levels of self-awareness we could speak to, and indeed, *I* can't tell one from the other without more examination of context.
I should go blather on my own blog to brain-dump a little better and get the hell back to my own work.
This all has me thinking out loud at the keys too much. Too many threads of thought that are a little unwoven right now, but I really appreciate this branching thread you kicked off. -
I should go blather on my own blog to brain-dump a little better and get the hell back to my own work.
This all has me thinking out loud at the keys too much. Too many threads of thought that are a little unwoven right now, but I really appreciate this branching thread you kicked off.@johannab Very kind of you to say so. Remember to like and subscribe
-
@johannab the AWS link was to showcase that even AWS itself can't prevent vibe-coding their mission-critical modules, and presumably a few skilled practitioners work there.
@johannab I guess I should concede that there are at least 2 people I know who actually use LLMs all the time and seem completely unaffected. They seem to be slightly more productive and produce normal-looking code with it. But they do not seem to possess any special insight; I have no idea what they're doing that's different.
-
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews (
) and the cognitive load of the automated comments was *enormous*.It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph I'd rather have it simply tell me what's wrong. (Or what it "thinks" is wrong.) Having to wade through AI code is like reviewing someone else's work, when you can't count on that person being at all competent. Best to just leave the coding to humans.
I'm all for AI finding faults; these can easily be checked for correctness. Infinitely harder for a human to check AI code for correctness. Which is all lost time against the schedule.
-
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews (
) and the cognitive load of the automated comments was *enormous*.It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph I have a friend who insists his AI partner writes great comments. I doubt that, and he's never provided an example. Since AI doesn't _understand_ the code, how can it write comments better than "We're going to loop through <thingies> and delete values out of range." Which the code already tells me. I want to know what you were _trying_ to do. The code may or may not do that, and comments which are based on the code can't help.
-
@bluewinds @janeishly @glyph I'd rather have it simply tell me what's wrong. (Or what it "thinks" is wrong.) Having to wade through AI code is like reviewing someone else's work, when you can't count on that person being at all competent. Best to just leave the coding to humans.
I'm all for AI finding faults; these can easily be checked for correctness. Infinitely harder for a human to check AI code for correctness. Which is all lost time against the schedule.
@agreeable_landfall @bluewinds @janeishly there's an alert fatigue problem there with LLM code review, but if I had to rank the harm it would definitely be lower down
-
@glyph my hypothesis on that is that, by virtue of literally being encodings of lexical fields and semantic proximity, and by virtue of their response being the logical continuation of the user's input, LLMs statistically pick up on and amplify subtle tendencies / biases in the user: if you feed it input that uses vocabulary and idioms semantically linked to low self-esteem, the model will more likely compute a reply with similar undertones, feeding said emotion. they amplify whatever emotion you put in, even accidentally.
(thread here: https://tech.lgbt/@nicuveo/116210599322080105 )@nicuveo seems plausible. I had a much vaguer hypothesis along these lines too. can’t dig up the toot right now but I definitely posted one a few weeks ago
