LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help.
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
Dare Obasanjo (@carnage4life@mas.to)
Attached: 1 image Hallucinations are the bane of my existence when using Claude Code and that has significantly improved after adding the following instructions to my Claude .MD file. Sharing for anyone else who uses Claude as a daily driver for analysis and writing but has gotten frustrated by it making things up.
mas.to (mas.to)
@benjamineskola
perhaps renaming them little lying malware might help ... -
@benjamineskola
perhaps renaming them little lying malware might help ...@Beatpoet13 in all seriousness I don’t like the terminology of ‘lying’ here either. It implies intent.
It’s not a lie for the same reason that it’s not a hallucination; there’s no difference from the LLM’s perspective. It’s not capable of evaluating the truth-value of its output, much less intentionally producing untrue (or true) statements. It’s mere probability.
Responsible usage of these tools would involve mechanisms to increase the probability of the desired output, but pretending it’s capable of evaluating that itself will not help at all.
-
@benjamineskola @solonovamax yeah it's just good at knowing what the next word is, so it can string something mathematically coherent
kind of like how "ai art" tends to be extremely generic looking because it quite literally aims to pick the most average in its dataset for a specific prompt
@nelson @benjamineskola @solonovamax
yeah, i think my take from about a year ago still mostly holds up https://biplus.social/@linkplay/114828181247605258 -
@solonovamax @benjamineskola i feel like it's more that it just wasn't built to give out answers, it was trained not to answer truthfully and "understand" but to just come up with something that sounds kind of convincing
@nelson @solonovamax @benjamineskola For better and worse, ML is an optimization algorithm designed to provide statistically close-to-ideal responses (with some jitter to break out of bad loops) to arbitrary input based on training (historic data). It's fantastic for, say, industrial control systems that want to keep a chemical reaction under control, but the nature of the math is that you can train it on any sequence of values, and this includes words. The problem is that language has contextual meaning, and the human brain is very much built to see patterns and meaning in things, even when they aren't there. Like how we see faces in clouds, for example. This technology is the faces in clouds engine.
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
Dare Obasanjo (@carnage4life@mas.to)
Attached: 1 image Hallucinations are the bane of my existence when using Claude Code and that has significantly improved after adding the following instructions to my Claude .MD file. Sharing for anyone else who uses Claude as a daily driver for analysis and writing but has gotten frustrated by it making things up.
mas.to (mas.to)
@benjamineskola It never stopped being this, just a version of this that has reduced errors. It's a corrective algorithm being fed noise and direction to correct towards. It has no sense of self, reality, or anything like that. Just an overgrown version of your noise canceling headphones algorithm, where the outside noise is it's starting point and your music is the prompt it tried to acheve.
-
@benjamineskola It never stopped being this, just a version of this that has reduced errors. It's a corrective algorithm being fed noise and direction to correct towards. It has no sense of self, reality, or anything like that. Just an overgrown version of your noise canceling headphones algorithm, where the outside noise is it's starting point and your music is the prompt it tried to acheve.
@MontgomeryGator I don’t think I said otherwise.
-
@nelson @solonovamax @benjamineskola For better and worse, ML is an optimization algorithm designed to provide statistically close-to-ideal responses (with some jitter to break out of bad loops) to arbitrary input based on training (historic data). It's fantastic for, say, industrial control systems that want to keep a chemical reaction under control, but the nature of the math is that you can train it on any sequence of values, and this includes words. The problem is that language has contextual meaning, and the human brain is very much built to see patterns and meaning in things, even when they aren't there. Like how we see faces in clouds, for example. This technology is the faces in clouds engine.
@nelson @solonovamax @benjamineskola If you're at all interested in some of the historic discourse regarding this sort of technology, google John Searle's Chinese room argument.
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
Dare Obasanjo (@carnage4life@mas.to)
Attached: 1 image Hallucinations are the bane of my existence when using Claude Code and that has significantly improved after adding the following instructions to my Claude .MD file. Sharing for anyone else who uses Claude as a daily driver for analysis and writing but has gotten frustrated by it making things up.
mas.to (mas.to)
@benjamineskola Thank you for putting it so clearly.
-
@MontgomeryGator I don’t think I said otherwise.
@benjamineskola Oh, I wasn't arguing with you. I was reinforcing your point with an engineering perspective. Literally why GenAI can never have hallucinations taken out, since its all just a controlled hallucination from the start.
-
@benjamineskola Oh, I wasn't arguing with you. I was reinforcing your point with an engineering perspective. Literally why GenAI can never have hallucinations taken out, since its all just a controlled hallucination from the start.
@MontgomeryGator ah, sorry, I didn’t realise that ‘this’ referred to the image and interpreted it as a disagreement with something I was saying.
No, you’re right, and that’s the problem with terminology like ‘hallucination’ or ‘lying’: the implication that there’s any distinction between the types of output it produces other than how much the user subjectively values them.
-
@Beatpoet13 in all seriousness I don’t like the terminology of ‘lying’ here either. It implies intent.
It’s not a lie for the same reason that it’s not a hallucination; there’s no difference from the LLM’s perspective. It’s not capable of evaluating the truth-value of its output, much less intentionally producing untrue (or true) statements. It’s mere probability.
Responsible usage of these tools would involve mechanisms to increase the probability of the desired output, but pretending it’s capable of evaluating that itself will not help at all.
@benjamineskola @Beatpoet13 it can be helpful to replace an llm with "repeatedly pressing the suggested word on your phone keyboard."
If it spits out "I am a funny hamster", you wouldn't say it lied.
Humans are just not conditioned --- not wired, frankly --- to engage with machine generated, syntactically valid text. We suck at it. The ELIZA Effect wins every time.
-
@benjamineskola @Beatpoet13 it can be helpful to replace an llm with "repeatedly pressing the suggested word on your phone keyboard."
If it spits out "I am a funny hamster", you wouldn't say it lied.
Humans are just not conditioned --- not wired, frankly --- to engage with machine generated, syntactically valid text. We suck at it. The ELIZA Effect wins every time.
@SheRaPantsuit @Beatpoet13 Yeah, there’s probably something to be said about UI affordances or something like that, where the chat interface guides people into assuming intentionality where there is none, and where some other presentation might be more objectively received.
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
Dare Obasanjo (@carnage4life@mas.to)
Attached: 1 image Hallucinations are the bane of my existence when using Claude Code and that has significantly improved after adding the following instructions to my Claude .MD file. Sharing for anyone else who uses Claude as a daily driver for analysis and writing but has gotten frustrated by it making things up.
mas.to (mas.to)
@benjamineskola@hachyderm.io well yeah if they knew how it worked they wouldnt advocate for it
- posted by Seraphine
Headmate Hopper -
@solonovamax @benjamineskola i feel like it's more that it just wasn't built to give out answers, it was trained not to answer truthfully and "understand" but to just come up with something that sounds kind of convincing
@nelson @benjamineskola agents (the general word for entities performing actions to achieve their goal, not talking about necessarily "AI agents", this word even applies to people, and even something like a thermostat that controls temperature) that wish to achieve their goals should be able to accurately model the real world
their ability to model the real world is directly correlated with their ability to achieve their goals. so, an agent which can accurately model the real world is able to achieve its goal much more easily that one that cannot accurately model the real worldand, people generally call an accurate model of the real world "truth"
hypothetically, the transformer architecture should be able to scale to human-level intelligence as it is turing-complete.
so, how it was trained doesn't necessarily matter, it's just that it is not capable of modeling the real world, so it cannot evaluate the truthiness of a statement -
@nelson @solonovamax @benjamineskola For better and worse, ML is an optimization algorithm designed to provide statistically close-to-ideal responses (with some jitter to break out of bad loops) to arbitrary input based on training (historic data). It's fantastic for, say, industrial control systems that want to keep a chemical reaction under control, but the nature of the math is that you can train it on any sequence of values, and this includes words. The problem is that language has contextual meaning, and the human brain is very much built to see patterns and meaning in things, even when they aren't there. Like how we see faces in clouds, for example. This technology is the faces in clouds engine.
@complexmath @nelson @solonovamax @benjamineskola
Exactly so. Ada Lovelace, patron saint of code, in the 1840s, gave us "Lady Lovelace’s Objection," whereupon she famously stated that machines "have no pretensions whatever to originate anything," saying they could only perform tasks they were instructed to do.
“AI” LLMs as they are sold to the rubes is just a spellchecker on steroids. It does not reason. It does not think. It correlates data it has been fed to reach a probability.
Telling it to not hallucinate is some serious cargo cult thinking.
-
@complexmath @nelson @solonovamax @benjamineskola
Exactly so. Ada Lovelace, patron saint of code, in the 1840s, gave us "Lady Lovelace’s Objection," whereupon she famously stated that machines "have no pretensions whatever to originate anything," saying they could only perform tasks they were instructed to do.
“AI” LLMs as they are sold to the rubes is just a spellchecker on steroids. It does not reason. It does not think. It correlates data it has been fed to reach a probability.
Telling it to not hallucinate is some serious cargo cult thinking.
@MissConstrue @complexmath @nelson @benjamineskola hypothetically it is possible for an artificial agent (read: "AI") to be capable of accurately modeling the world and "thinking", however it seems that this is not currently even remotely the case.
-
LLM advocates still don’t seem to be able to comprehend that ordering the machine not to ‘make stuff up’ doesn’t help. It doesn’t know when it’s making stuff up, and it couldn’t change that even if you told it to. (In fact it’s always just making stuff up, and is only ever true by chance.)
Part of why I’m so negative about them is that their advocates simply do not understand how they work and do not seem to want to.
Dare Obasanjo (@carnage4life@mas.to)
Attached: 1 image Hallucinations are the bane of my existence when using Claude Code and that has significantly improved after adding the following instructions to my Claude .MD file. Sharing for anyone else who uses Claude as a daily driver for analysis and writing but has gotten frustrated by it making things up.
mas.to (mas.to)
@benjamineskola I am a safety engineer for safety-relevant and mission-critical systems. And it is disheartening to see safety professionals at international conferences present 2-page-long prompts, doing basically all this, but much more so, and expect their "spicy autocomplete machine" (@pluralistic) to create safety analyses this way. And always talking about the LLM as if it could think. And prompt it to show its internal steps and "reasoning", not understanding that it does no such thing. It just creates another string of words that sounds as if an intelligence were describing the inner workings.
The upshot almost always is "It sucks, we have to check and correct everything. We love it. It is the future!"
🤪 -
@MissConstrue @complexmath @nelson @benjamineskola hypothetically it is possible for an artificial agent (read: "AI") to be capable of accurately modeling the world and "thinking", however it seems that this is not currently even remotely the case.
@solonovamax @MissConstrue @complexmath @nelson I would not expect a large language model to be capable of doing so, no matter how advanced. An ‘AI’ ‘agent’ based on some other technology? Perhaps. But at that point we’re literally just saying ‘technically it’s not impossible for this to exist in future’; we’re in the realm of science fiction.
-
@junkman I see your car analogy and I raise you a slavery analogy: that, too, had “proved some degree of usefulness” for “some people very good at wielding it as a tool”.
And I bet you’d rather not know how it worked (works) if you were the one finding it handy for the benefits it provided you. Saying you don’t partake while loudly proclaiming its usefulness is not fooling anyone.
@mushroom_man that analogy is also useful.
I don't like that the whole LLM functionality is based on stealing humanity's knowledge for profit. Outright violated (shitty) copyright law and basically the regular people got screwed over while mega corporations just agreed not to make a big fuzz about it.
The foundations are so rotten and just so people can chat with their computers instead of doing manual pointing and typing.
-
@SheRaPantsuit @Beatpoet13 Yeah, there’s probably something to be said about UI affordances or something like that, where the chat interface guides people into assuming intentionality where there is none, and where some other presentation might be more objectively received.
@benjamineskola @SheRaPantsuit @Beatpoet13
It's even worse. Interactive LLMs create a linguistic bypass channel that "connects" parts in our minds/brains that are ordinarily separated by filters for plausibility and attenuating uncontrolled feedback. Furthermore, they can be tailored to adversely amplify select thought patterns.
They're the first, rudimentary implementation of the kind of cognitohazards that used to be science fiction.
Already they're potent cult-indoctrination machines.
1/
