"Just try whatever open weight small LLM model with "thinking" or "reasoning" or whatever they market it as"

That's what they market it as, but it's not what it's actually doing. Everything that it generates is a story. They are not showing you "what is going on under the hood", they are writing a story.

Any text it generates that says things like "the table does not exist in the schema, so it is probably part of an old project and is no longer relevant" or "NEVER FUCKING GUESS!” – and that’s exactly what I did." is not telling you anything about the process the LLM went through, it is recreating a story about what a hypothetical human might have done.

The "reasoning steps" that you are writing about don't actually exist.

"LLM "reasoning", not shown to the user but still part of the context because that's how "thinking" agents work:"

No, that's not how "thinking agents work". That's how humans work, but not how large language models work. Large language models don't think "the table is part of an old project", they don't "know" what "old" or "project" mean. They match the request against similar requests and generate similar code.

That is context in the prompt, not in the source text that created the model that you are asking the question "why did you do X".

The answer you get is from that source corpus, and contains lots of text about what a human might do, but the LLM doesn't do anything for those reasons.

The "why" of "why did you do X" is always "because those were the next likely tokens" and never anything related to "what would a human say if you asked them".

"You're assuming that there is no other context provided with the question, and that the training does not take into account that context. "

Well, yes, I am assuming that. Because the question is "why did you do this thing that nobody expected you to do". The context-specific answer that you *need* is far too nuanced and unpredictable to possibly be explicitly in the training data.

"Why" is definitely a word from the training data, and "why did you do that?" is definitely also part of things asked a lot, that OpenAI and others have trained on,"

Yes, and the text that follows is an answer to *a different situation*, and so it's basically fanfic about itself. That's all it can ever produce when you ask it "why". Fanfic.

I absolutely have. I keep this in mind ALL THE TIME when I test these things and EVERY TIME they can trivially be led into generating pure nonsense by exploiting that fact.

"the LLM has no semantic model of reality, only a surface statistical model of language present in the training data."

Absolutely this.

Deliberately so. LLMs are the end result of 50 years of cynical software developers trying to "beat the Turing test". They are automated gaslighting.

can you two take your semantics argument elsewhere; I am not interested in philosophical horseshit when there are specific, practical considerations that are causing specific, enumerable harms.

Quentin Tarantino would not think so.

no.

> One of the worst mistakes the opposition can make is extending contempt for the tyrant into contempt for the tyrant’s supporters. Most of these supporters sincerely believed that the tyrant would be more likely to solve their problems — often real grievances that the opposition had failed to address. Blaming the supporters denies the reality of the failures and reinforces their support for the tyrant.

@munin

My view *might* be partially influenced by a quote from this piece on "The Rise and Fall of Petty Tyrants" (quote in next message)

The Rise & Fall Of ‘Petty Tyrants’ | NOEMA

History shows that bad leaders can successfully undermine democracy — but the story always ends the same way.

NOEMA (www.noemamag.com)

@munin