I wish I could recommend this piece more, because it makes a bunch of great points, but the "normal technology" case feels misleading to me.
-
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews (
) and the cognitive load of the automated comments was *enormous*.It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph The "tarpit for thinking" framing is perfect. AI code review that flags things but suggests wrong fixes is worse than no review at all — it steals your attention for nothing.
That's why we went a different direction with our scanner. Instead of reviewing individual code changes, we check structural signals: does CI exist? Are there tests? Are secrets exposed? Binary yes/no checks that don't require you to evaluate AI-generated suggestions. repofortify.com
-
Furthermore, it is not "nuts" to dismiss the experience of an LLM user. In fact, you must dismiss all experiences of LLM users, even if the LLM user is yourself. Fly by instruments because the cognitive fog is too think for your eyes to see.
Because the interesting, novel thing about LLMs, the thing that makes them dangerous and interesting, is that they are, by design, epistemic disruptors.
They can produce symboloids more rapidly than any thinking mind. Repetition influences cognition.
@glyph "They can produce symboloids more rapidly than a thinking mind" maybe if someone thinks really slowly? either that or there‘s some much faster llm i‘ve never heard of
i find the output on these infuriating because they generate slower than i read, so when i have to test them for whatever reason (usually to show how comically poorly it does at a given application as an example of why not to use it) i have to scroll up until i think its done generating before reading
-
More to the point though in this metaphor where you're getting a potentially-infected scrape at work, we are living in the pre-germ-theory age of AI. We are aware that it might be dangerous sometimes, but we don't know to whom or why. We are attempting to combat miasma with bloodletting right now, and putting the miasma-generator in every home before we know what it's actually doing.
@glyph potentially an even better metaphor is RSI, though that does lead to the "you're holding it wrong" argument which isn't applicable, but incidental injuries are in the same bucket but it's just less obvious.
-
@johannab I guess I should concede that there are at least 2 people I know who actually use LLMs all the time and seem completely unaffected. They seem to be slightly more productive and produce normal-looking code with it. But they do not seem to possess any special insight; I have no idea what they're doing that's different.
@glyph I think there are a lot of individual, and small-scale social factors, that make a huge difference here.
Prior domain expertise, personal self-image, ability to separate work and not-work life, other social anchors in the non-digital world ... I feel like these all have an interaction.
I'm really concerned at what I see of students, even grad students around me, who have basically not *learned* a thing about life without these.
-
@glyph I think there are a lot of individual, and small-scale social factors, that make a huge difference here.
Prior domain expertise, personal self-image, ability to separate work and not-work life, other social anchors in the non-digital world ... I feel like these all have an interaction.
I'm really concerned at what I see of students, even grad students around me, who have basically not *learned* a thing about life without these.
@glyph Less concerned about say, my spouse, who had 28 years sysadmin experience behind him when his hype-chasing CEO declared that All Shalt Use the AI Or Suffer The Performance Review Consequences.
He basically dictated what he otherwise would have scripted and let the clanker write the scripts. I'm not sure it saved much time, but he's found a couple of spots where it extracted something he hadn't thought of and got past a sticking point.
-
@glyph Less concerned about say, my spouse, who had 28 years sysadmin experience behind him when his hype-chasing CEO declared that All Shalt Use the AI Or Suffer The Performance Review Consequences.
He basically dictated what he otherwise would have scripted and let the clanker write the scripts. I'm not sure it saved much time, but he's found a couple of spots where it extracted something he hadn't thought of and got past a sticking point.
@johannab I have not done a comprehensive survey, but I simultaneously believe that A) you're directionally correct and the relevant factors are *something* like this, and B) there are some counterexamples where very well-adjusted, experienced, emotionally regulated people suddenly and unpredictably lurch off into the deep end, so there's something non-obvious going on too.
-
@atax1a looping back to some of Cory’s good points here (from another essay): it’s a picture-perfect example of reverse-centaur accountability-sink logic. their jobs are about to become *profoundly* miserable
-
@violetmadder have you read https://blog.glyph.im/2025/08/futzing-fraction.html ? I use this metaphor a lot.
-
@johannab I have not done a comprehensive survey, but I simultaneously believe that A) you're directionally correct and the relevant factors are *something* like this, and B) there are some counterexamples where very well-adjusted, experienced, emotionally regulated people suddenly and unpredictably lurch off into the deep end, so there's something non-obvious going on too.
@glyph oh, there certainly is!
Human brains tangle many aspects of identity into our relational constructs, including in the working world. Those interact with our cognitive and neurological processes, and create minds that are as unique as fingerprints.
Its as complex, if not more so, than something like addiction. How does one person regularly use THC to be free from chronic pain with no apparent side effects, but another smokes up once and suffers a psychotic break?
-
@bluewinds @delta_vee @kirakira @glyph I don't think the analogies are good because asbestos is a fantastic insulator, lead is a really helpful additive for petrol and makes fantastic pigments and is really convenient for piping... and the hidden side-effects are the problem. Whereas LLMs _don't_ deliver that primary benefit
LLMs are more like... cheap laminate flooring, produced with wood pulp harvested unsustainably from old-growth forests and made by grossly exploited factory workers overseas... superficially convenient when remodelling your kitchen and rapidly ubiquitous but also quite unsatisfying and a right faff to work around once it's established
@jackeric @bluewinds @delta_vee @kirakira @glyph Only half seriously, but therefore also not totally *unseriously*:
Zip fuel.
If you've never heard of it, there's good reason. But it did attempt to address a legitimate concern at the time: getting more power out of a given volume of jet fuel. Just put highly reactive boron compounds in it. Specifically, *pyrophoric* boron compounds, which don't even need high heat to ignite.
The fuel did indeed produce more power, but it was very toxic (both in raw form and after combustion), and it seriously corroded jet engine parts, leading to an enormous maintenance headache for any aircraft that tried to use the fuel.
(Maintenance headaches ... sound familiar?)
A very good example of what can happen when you decide "speed of one specific component", whether airplane flights or writing code, overwhelmingly dominates a thought process.
-
@glyph oh, there certainly is!
Human brains tangle many aspects of identity into our relational constructs, including in the working world. Those interact with our cognitive and neurological processes, and create minds that are as unique as fingerprints.
Its as complex, if not more so, than something like addiction. How does one person regularly use THC to be free from chronic pain with no apparent side effects, but another smokes up once and suffers a psychotic break?
@glyph No human psyche is as obvious and superficial as its audible or textual outputs. Nor as robust.
There are so many systems interconnecting when we interact with other brains. When we can manipulate or corrupt a simulated one which has already connected with organic ones, there's no single, clear pathway of cause-and-effect.
We're trying to unscramble the eggs.
