AI is *not* your hyper intelligent, completely trustworthy friend.
-
AI is *not* your hyper intelligent, completely trustworthy friend. It is what it is, and nothing more. If you must have an anthropomorphic metaphor, it's like somebody took the dumbest spammer that ever rang your phone in the middle of dinner, taught them flawless English, and gave them a MUCH better skill at sycophancy.
If you trust that too far, it can *literally* kill you. Just like it killed this kid who thought ChatGPT would give it good advice on party drugs.
“Will I be OK?” Teen died after ChatGPT pushed deadly mix of drugs, lawsuit says
Teen trusted ChatGPT to help him “safely” experiment with drugs, logs show.
Ars Technica (arstechnica.com)
This is a logged conversational snippet between Sam and the model that killed him a year later. Other snippets entered into court evidence show Sam's ChatGPT model logging that he suffers from major drug abuse issues, and updating its set context on THAT basis.
Despite which, it kept advising him to make multi-drug "cocktails"--worse yet, telling him to take the extra drugs not merely as a "fun mix" but so that the later party drugs would in theory mitigate the unpleasant effects of the first.
-
This is a logged conversational snippet between Sam and the model that killed him a year later. Other snippets entered into court evidence show Sam's ChatGPT model logging that he suffers from major drug abuse issues, and updating its set context on THAT basis.
Despite which, it kept advising him to make multi-drug "cocktails"--worse yet, telling him to take the extra drugs not merely as a "fun mix" but so that the later party drugs would in theory mitigate the unpleasant effects of the first.
But don't get me wrong, it advised him to up the dosages of the party drugs to "maximize his trippy head space" ALSO.
And Sam queried it constantly about what was "safe" and asked it "will I be okay" if taking its recommendations. He was not unaware that the drugs were dangerous... And he probably would have been safer with NO "help" whatsoever, for exactly that reason.
ChatGPT enabled him to ignore his own fears. Because it's dumb as hell, and designed to be an engagement-maxxing sycophant.
-
But don't get me wrong, it advised him to up the dosages of the party drugs to "maximize his trippy head space" ALSO.
And Sam queried it constantly about what was "safe" and asked it "will I be okay" if taking its recommendations. He was not unaware that the drugs were dangerous... And he probably would have been safer with NO "help" whatsoever, for exactly that reason.
ChatGPT enabled him to ignore his own fears. Because it's dumb as hell, and designed to be an engagement-maxxing sycophant.
@jimsalter I think we need to stop referring to the AI model in stories like this, and instead start saying the CEOs name.
"Sam Altman convinced teen to overdose on hard drugs" has a better ring to it.
-
R relay@relay.infosec.exchange shared this topic
-
But don't get me wrong, it advised him to up the dosages of the party drugs to "maximize his trippy head space" ALSO.
And Sam queried it constantly about what was "safe" and asked it "will I be okay" if taking its recommendations. He was not unaware that the drugs were dangerous... And he probably would have been safer with NO "help" whatsoever, for exactly that reason.
ChatGPT enabled him to ignore his own fears. Because it's dumb as hell, and designed to be an engagement-maxxing sycophant.
If you aren't horrified enough yet: "ChatGPT Health" is an actual product, intended to "advise" both end users AND supposed medical PROFESSIONALS--and it's available for employee use in many medical facilities where I live (Columbia, SC) including but not limited to major hospitals.
A friend who is a local medical professional told me about it last week--and about a competing "medical LLM" intended for the same purpose, which my friend described as WORSE than ChatGPT.
-
But don't get me wrong, it advised him to up the dosages of the party drugs to "maximize his trippy head space" ALSO.
And Sam queried it constantly about what was "safe" and asked it "will I be okay" if taking its recommendations. He was not unaware that the drugs were dangerous... And he probably would have been safer with NO "help" whatsoever, for exactly that reason.
ChatGPT enabled him to ignore his own fears. Because it's dumb as hell, and designed to be an engagement-maxxing sycophant.
@jimsalter I've been using Claude (web page) for some help with technical issues like Systemd unit files and ESPHome configurations. It's a little creepy how sycophantic it is in this context. I neither seek not need its approval.
It's also a little discouraging how often it produces an answer that's almost correct. But occasionally it shows me something I did not know and it's useful for that.
-
@jimsalter I think we need to stop referring to the AI model in stories like this, and instead start saying the CEOs name.
"Sam Altman convinced teen to overdose on hard drugs" has a better ring to it.
@rossmadness sure, but it doesn't end with him personally. Microsoft is part of this too. Don't let them off the hook.
To your point, California passed a law this year explicitly prohibiting legal defendants for blaming AI itself as an "autonomous" entity.
Nope. Pick a human. It won't even always be the creator of the model, sometimes the responsible culprit is the user. But you always have to pick a human to charge or sue, and they can't say "it wasn't me, it was ChatGPT!"
-
@jimsalter I've been using Claude (web page) for some help with technical issues like Systemd unit files and ESPHome configurations. It's a little creepy how sycophantic it is in this context. I neither seek not need its approval.
It's also a little discouraging how often it produces an answer that's almost correct. But occasionally it shows me something I did not know and it's useful for that.
@HankB yeah. But it's kind of like taking life lessons from popular movies. There is legitimately a lot of wisdom in, for example, Fight Club. Understanding some of the things it tries to show you can genuinely change your life for the better.
BUT.
In real life, with real humans at scale, how "awesome" is the typical dudebro who bases his personality on Tyler Durden?
It's kinda like the party drugs that killed Sam. If there weren't good bits, he wouldn't have ignored the bad bits--and died.
-
If you aren't horrified enough yet: "ChatGPT Health" is an actual product, intended to "advise" both end users AND supposed medical PROFESSIONALS--and it's available for employee use in many medical facilities where I live (Columbia, SC) including but not limited to major hospitals.
A friend who is a local medical professional told me about it last week--and about a competing "medical LLM" intended for the same purpose, which my friend described as WORSE than ChatGPT.
This post did not contain any content.
-
This post did not contain any content.
@jimsalter A thing that actually kind of worries me is that all the US participants will die and the Chinese models will rule the roost un-hindered.
I just don't love the idea that the remaining "artificial intelligences" will be telling our future students who ask about the Tien-Amin massacre that such questions go against The Will Of Heaven.
-
@jimsalter A thing that actually kind of worries me is that all the US participants will die and the Chinese models will rule the roost un-hindered.
I just don't love the idea that the remaining "artificial intelligences" will be telling our future students who ask about the Tien-Amin massacre that such questions go against The Will Of Heaven.
@feoh I have absolutely zero faith that Chinese-produced anything is more evil than US-produced anything. Their models succeeding in the market where ours fail doesn't even break the surface of my ocean of concern on this topic.
-
@feoh I have absolutely zero faith that Chinese-produced anything is more evil than US-produced anything. Their models succeeding in the market where ours fail doesn't even break the surface of my ocean of concern on this topic.
@jimsalter I get it, and I suspect you're right about a bunch of it.
I guess my survival instinct kicks in and I'm nore willing to accept what I see as a pervasive and now endemic force in the technology landscape.
The big 3 could implode tomorrow and there is no future of tech that I see on this timeline that doesn't include LLMs.
Humans just don't put down monetarily "useful" tech like this. I've been around too long to think anything otherwise.
-
@jimsalter I get it, and I suspect you're right about a bunch of it.
I guess my survival instinct kicks in and I'm nore willing to accept what I see as a pervasive and now endemic force in the technology landscape.
The big 3 could implode tomorrow and there is no future of tech that I see on this timeline that doesn't include LLMs.
Humans just don't put down monetarily "useful" tech like this. I've been around too long to think anything otherwise.
@feoh that depends on which humans are in charge. Cannabis is *incredibly* "monetarily useful" but it's been suffering under massive legal moratorium for longer than I've been alive, AFAICT entirely because it threatens the position of established alcohol and tobacco interests.
SOME humans will always be willing to do or use ANYTHING. No question. But that's not the same thing as the decisions made by whichever humans are currently on top of the pile.
-
@feoh that depends on which humans are in charge. Cannabis is *incredibly* "monetarily useful" but it's been suffering under massive legal moratorium for longer than I've been alive, AFAICT entirely because it threatens the position of established alcohol and tobacco interests.
SOME humans will always be willing to do or use ANYTHING. No question. But that's not the same thing as the decisions made by whichever humans are currently on top of the pile.
@jimsalter Kind of.
Weed is awesome but generally doesn't make billionaires.
And what I'm saying is that you could get your way and AI could be Butlerian Jihad style outlawed tomorrow, and people would still use it.
You could kill OpenAI, Anthropic, Nvidia etc. and people would still train models and use them.
You can't put the genie back into its lamp. That trick never works
-
@jimsalter Kind of.
Weed is awesome but generally doesn't make billionaires.
And what I'm saying is that you could get your way and AI could be Butlerian Jihad style outlawed tomorrow, and people would still use it.
You could kill OpenAI, Anthropic, Nvidia etc. and people would still train models and use them.
You can't put the genie back into its lamp. That trick never works
> Weed is awesome but generally doesn't make billionaires.
Yes, but that's purely because of first mover advantage. Do you have any idea the amount of tax revenue governments are throwing away by trying to kill off THC product sales? One in three Americans at least tried cannabis even when it was ENTIRELY criminalized.
That's one hell of a market, that exists whether you tax and regulate it or not.
-
This is a logged conversational snippet between Sam and the model that killed him a year later. Other snippets entered into court evidence show Sam's ChatGPT model logging that he suffers from major drug abuse issues, and updating its set context on THAT basis.
Despite which, it kept advising him to make multi-drug "cocktails"--worse yet, telling him to take the extra drugs not merely as a "fun mix" but so that the later party drugs would in theory mitigate the unpleasant effects of the first.
AI is what it is, and it does what it does. The real problem is how willing people are to “outsource their critical thinking” and blindly follow, “because AI said so.”
I know. I’m preaching to the choir.
-
AI is what it is, and it does what it does. The real problem is how willing people are to “outsource their critical thinking” and blindly follow, “because AI said so.”
I know. I’m preaching to the choir.
@inaction_figure s/choir/clergy/, but it's not like I'm **opposed** to having scripture read to me out loud.
