I wish I could recommend this piece more, because it makes a bunch of great points, but the "normal technology" case feels misleading to me.
-
@glyph
Here's an industrial accident that's easy to miss:A hydraulic fluid line bursts while you're working on a machine, injecting toxic and/or hot liquid under your skin at high pressure.
https://en.wikipedia.org/wiki/High_pressure_injection_injury
"Although the initial wound often seems minor, the unseen, internal damage can be severe. With hydraulic fluids, paint, and detergents, these injuries are extremely serious as most hydraulic fluids and organic solvents are highly toxic."@dec23k okay definitely not clicking on that link, yeesh
-
Having read over Doctorow's rant-du-jour twice now, I do think when he said "they" were not vibe coding mission-critial AWS modules", he was referring to the "they" in the previous paragraph, being developers he's spoken to, some of whom were friends he knows well.
So.... could be very differently skilled people from "some hack in a code assembly shop driving at a reckless pace because Amazon stock needs a bump".
It's all back to, though, defining "AI".
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
-
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
@johannab So yes, maybe his contacts are transcendentally better programmers than mine, and they've ascended to a plane of subjective self-assessment beyond mere mortals, but if they're anything like the (extremely skilled, extremely experienced) people I've watched fall into this trap, I'm highly skeptical
-
@johannab So yes, maybe his contacts are transcendentally better programmers than mine, and they've ascended to a plane of subjective self-assessment beyond mere mortals, but if they're anything like the (extremely skilled, extremely experienced) people I've watched fall into this trap, I'm highly skeptical
@johannab the AWS link was to showcase that even AWS itself can't prevent vibe-coding their mission-critical modules, and presumably a few skilled practitioners work there.
-
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
@glyph Fair, for sure.
I just realized when reading it over that was a spot there could be a disconnect between the "they" being referred to in the essay narrative as written.
I feel like my immediate, 1-degree friends, acquaintances and colleagues include amongst them all the theoretical levels of self-awareness we could speak to, and indeed, *I* can't tell one from the other without more examination of context.
-
@glyph Fair, for sure.
I just realized when reading it over that was a spot there could be a disconnect between the "they" being referred to in the essay narrative as written.
I feel like my immediate, 1-degree friends, acquaintances and colleagues include amongst them all the theoretical levels of self-awareness we could speak to, and indeed, *I* can't tell one from the other without more examination of context.
I should go blather on my own blog to brain-dump a little better and get the hell back to my own work.
This all has me thinking out loud at the keys too much. Too many threads of thought that are a little unwoven right now, but I really appreciate this branching thread you kicked off. -
I should go blather on my own blog to brain-dump a little better and get the hell back to my own work.
This all has me thinking out loud at the keys too much. Too many threads of thought that are a little unwoven right now, but I really appreciate this branching thread you kicked off.@johannab Very kind of you to say so. Remember to like and subscribe
-
@johannab the AWS link was to showcase that even AWS itself can't prevent vibe-coding their mission-critical modules, and presumably a few skilled practitioners work there.
@johannab I guess I should concede that there are at least 2 people I know who actually use LLMs all the time and seem completely unaffected. They seem to be slightly more productive and produce normal-looking code with it. But they do not seem to possess any special insight; I have no idea what they're doing that's different.
-
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews (
) and the cognitive load of the automated comments was *enormous*.It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph I'd rather have it simply tell me what's wrong. (Or what it "thinks" is wrong.) Having to wade through AI code is like reviewing someone else's work, when you can't count on that person being at all competent. Best to just leave the coding to humans.
I'm all for AI finding faults; these can easily be checked for correctness. Infinitely harder for a human to check AI code for correctness. Which is all lost time against the schedule.
-
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews (
) and the cognitive load of the automated comments was *enormous*.It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph I have a friend who insists his AI partner writes great comments. I doubt that, and he's never provided an example. Since AI doesn't _understand_ the code, how can it write comments better than "We're going to loop through <thingies> and delete values out of range." Which the code already tells me. I want to know what you were _trying_ to do. The code may or may not do that, and comments which are based on the code can't help.
-
@bluewinds @janeishly @glyph I'd rather have it simply tell me what's wrong. (Or what it "thinks" is wrong.) Having to wade through AI code is like reviewing someone else's work, when you can't count on that person being at all competent. Best to just leave the coding to humans.
I'm all for AI finding faults; these can easily be checked for correctness. Infinitely harder for a human to check AI code for correctness. Which is all lost time against the schedule.
@agreeable_landfall @bluewinds @janeishly there's an alert fatigue problem there with LLM code review, but if I had to rank the harm it would definitely be lower down
-
@glyph my hypothesis on that is that, by virtue of literally being encodings of lexical fields and semantic proximity, and by virtue of their response being the logical continuation of the user's input, LLMs statistically pick up on and amplify subtle tendencies / biases in the user: if you feed it input that uses vocabulary and idioms semantically linked to low self-esteem, the model will more likely compute a reply with similar undertones, feeding said emotion. they amplify whatever emotion you put in, even accidentally.
(thread here: https://tech.lgbt/@nicuveo/116210599322080105 )@nicuveo seems plausible. I had a much vaguer hypothesis along these lines too. can’t dig up the toot right now but I definitely posted one a few weeks ago
-
@glyph @nils_berger
this study argues that it encourages cognitive outsourcing on a new level, which in long term period could result in getting used to less cognitive activity, at least for certain tasks.link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6097646
@bbacc thank you!
-
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews (
) and the cognitive load of the automated comments was *enormous*.It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph The "tarpit for thinking" framing is perfect. AI code review that flags things but suggests wrong fixes is worse than no review at all — it steals your attention for nothing.
That's why we went a different direction with our scanner. Instead of reviewing individual code changes, we check structural signals: does CI exist? Are there tests? Are secrets exposed? Binary yes/no checks that don't require you to evaluate AI-generated suggestions. repofortify.com
-
Furthermore, it is not "nuts" to dismiss the experience of an LLM user. In fact, you must dismiss all experiences of LLM users, even if the LLM user is yourself. Fly by instruments because the cognitive fog is too think for your eyes to see.
Because the interesting, novel thing about LLMs, the thing that makes them dangerous and interesting, is that they are, by design, epistemic disruptors.
They can produce symboloids more rapidly than any thinking mind. Repetition influences cognition.
@glyph "They can produce symboloids more rapidly than a thinking mind" maybe if someone thinks really slowly? either that or there‘s some much faster llm i‘ve never heard of
i find the output on these infuriating because they generate slower than i read, so when i have to test them for whatever reason (usually to show how comically poorly it does at a given application as an example of why not to use it) i have to scroll up until i think its done generating before reading
-
More to the point though in this metaphor where you're getting a potentially-infected scrape at work, we are living in the pre-germ-theory age of AI. We are aware that it might be dangerous sometimes, but we don't know to whom or why. We are attempting to combat miasma with bloodletting right now, and putting the miasma-generator in every home before we know what it's actually doing.
@glyph potentially an even better metaphor is RSI, though that does lead to the "you're holding it wrong" argument which isn't applicable, but incidental injuries are in the same bucket but it's just less obvious.
-
@johannab I guess I should concede that there are at least 2 people I know who actually use LLMs all the time and seem completely unaffected. They seem to be slightly more productive and produce normal-looking code with it. But they do not seem to possess any special insight; I have no idea what they're doing that's different.
@glyph I think there are a lot of individual, and small-scale social factors, that make a huge difference here.
Prior domain expertise, personal self-image, ability to separate work and not-work life, other social anchors in the non-digital world ... I feel like these all have an interaction.
I'm really concerned at what I see of students, even grad students around me, who have basically not *learned* a thing about life without these.
-
@glyph I think there are a lot of individual, and small-scale social factors, that make a huge difference here.
Prior domain expertise, personal self-image, ability to separate work and not-work life, other social anchors in the non-digital world ... I feel like these all have an interaction.
I'm really concerned at what I see of students, even grad students around me, who have basically not *learned* a thing about life without these.
@glyph Less concerned about say, my spouse, who had 28 years sysadmin experience behind him when his hype-chasing CEO declared that All Shalt Use the AI Or Suffer The Performance Review Consequences.
He basically dictated what he otherwise would have scripted and let the clanker write the scripts. I'm not sure it saved much time, but he's found a couple of spots where it extracted something he hadn't thought of and got past a sticking point.
-
@glyph Less concerned about say, my spouse, who had 28 years sysadmin experience behind him when his hype-chasing CEO declared that All Shalt Use the AI Or Suffer The Performance Review Consequences.
He basically dictated what he otherwise would have scripted and let the clanker write the scripts. I'm not sure it saved much time, but he's found a couple of spots where it extracted something he hadn't thought of and got past a sticking point.
@johannab I have not done a comprehensive survey, but I simultaneously believe that A) you're directionally correct and the relevant factors are *something* like this, and B) there are some counterexamples where very well-adjusted, experienced, emotionally regulated people suddenly and unpredictably lurch off into the deep end, so there's something non-obvious going on too.
-
@atax1a looping back to some of Cory’s good points here (from another essay): it’s a picture-perfect example of reverse-centaur accountability-sink logic. their jobs are about to become *profoundly* miserable
