TIL that saying "holy shit don't use ChatGPT for medical advice" is a "purity test".
-
TIL that saying "holy shit don't use ChatGPT for medical advice" is a "purity test". i didn't know that before. in fact I still don't.
@davidgerard this one hit close to my heart because I’ve had two family members die in large part because their caretaker ignored medical advice and used awful alternative medicine information from the internet to try and treat them.
an LLM can’t do critique. as you’ve said, truth is not a data type in an LLM. all of these models suck in every form of medical crankery available on the internet, mix it with words from authentic medical sources, and present it all as credible.
-
@davidgerard this one hit close to my heart because I’ve had two family members die in large part because their caretaker ignored medical advice and used awful alternative medicine information from the internet to try and treat them.
an LLM can’t do critique. as you’ve said, truth is not a data type in an LLM. all of these models suck in every form of medical crankery available on the internet, mix it with words from authentic medical sources, and present it all as credible.
@davidgerard I know that alternative medicine has a body count; I’ve seen it in the flesh. I know what some of the horseshit on the Internet can do if you’re very desperate or very trusting.
the LLM lowers the trust barrier because the crank information is no longer crank flavored, but it’s still dangerous as fuck to follow the advice.
I keep seeing LLMs be presented as better than nothing and that’s wrong. I wish the people who needed help could get it, but the LLM is worse than nothing.
-
@davidgerard I am pretty sure that OpenAI do not have a license to practice medicine and are not a (human) member of the BMA so by giving medical advice they (the humans responsible for the software) are potentially committing an imprisonable offense ...
@cstross @davidgerard who will you imprison? The ceo? The programmers? The qa team?
One of the big draws of tech is the ability to turn human error (and malfeasance) into "computer error". And society has been trained to believe software errors aren't anyone's fault so there's no one to hold accountable
That needs to change. Companies need to be accountable for their "computer errors" - especially when they're baked into design and not actually errors
-
TIL that saying "holy shit don't use ChatGPT for medical advice" is a "purity test". i didn't know that before. in fact I still don't.
@davidgerard I don't even know what that means. I'm referring to "purity test".
-
Hmm, and presumably anyone operating a general-purpose chatbot that could conceivably be prompted to give such advice (e.g. as the conversational interface to a regular web-page) are also plausibly at risk?
@dwm @davidgerard Yes, although it all depends on whether the GMC (and the Police) have the guts to go after a large foreign corporation with deep pockets. It probably won't happen unless there's a major death-related scandal and/or one of the aforementioned corporations decides to go after the competition, i.e. small locally run and/or open source models with broad training sets.
-
@cstross @davidgerard Any coin can give medical advice. I just ask the coin: should I take this medicine, say "head". Then I throw the coin. I hope the people at the coin minting facility get imprisoned for that.
@waffelhard @cstross @davidgerard …as coins are often claimed to be able to replace doctors by the coin minting industry and its adherents.
-
@cstross @davidgerard who will you imprison? The ceo? The programmers? The qa team?
One of the big draws of tech is the ability to turn human error (and malfeasance) into "computer error". And society has been trained to believe software errors aren't anyone's fault so there's no one to hold accountable
That needs to change. Companies need to be accountable for their "computer errors" - especially when they're baked into design and not actually errors
@Jer @cstross @davidgerard it's the CEOs job to manage legal risk. Imprison the CEO.
-
@Jer @cstross @davidgerard it's the CEOs job to manage legal risk. Imprison the CEO.
@wronglang @cstross @davidgerard I actually agree. It would certainly justify the vast amounts of money they make if they had to take personal responsibility for their harmful decisions. Might make them think a little harder about their decisions
-
@cstross @davidgerard who will you imprison? The ceo? The programmers? The qa team?
One of the big draws of tech is the ability to turn human error (and malfeasance) into "computer error". And society has been trained to believe software errors aren't anyone's fault so there's no one to hold accountable
That needs to change. Companies need to be accountable for their "computer errors" - especially when they're baked into design and not actually errors
@Jer @davidgerard That's a broader corporate liability question. Personally I'd LIKE to see the C-suite and boards of corporations that kill people sentenced to serious prison time. (Lower level staff too, but only if it's found that they made decisions that led to deaths on their own initiative. The directors *are responsible for the company's actions*.)
Going further: the current privileged legal status of corporations is an obscenity and needs to be de-legitimized.
-
@davidgerard I know that alternative medicine has a body count; I’ve seen it in the flesh. I know what some of the horseshit on the Internet can do if you’re very desperate or very trusting.
the LLM lowers the trust barrier because the crank information is no longer crank flavored, but it’s still dangerous as fuck to follow the advice.
I keep seeing LLMs be presented as better than nothing and that’s wrong. I wish the people who needed help could get it, but the LLM is worse than nothing.
@davidgerard LLMs get alternative medicine patients to the “I don’t care what you say, *I* feel better” point of no return so much quicker because they don’t know it’s alternative medicine. some of it might even be legitimate medicine that works! and all this does is make them less skeptical until they get output that’s plausible but fatal, or until the damage from what they’ve been doing builds up and they can’t survive anymore. and thanks to the LLM, they’ll fight off anyone who tries to help.
-
@davidgerard LLMs get alternative medicine patients to the “I don’t care what you say, *I* feel better” point of no return so much quicker because they don’t know it’s alternative medicine. some of it might even be legitimate medicine that works! and all this does is make them less skeptical until they get output that’s plausible but fatal, or until the damage from what they’ve been doing builds up and they can’t survive anymore. and thanks to the LLM, they’ll fight off anyone who tries to help.
@zzt @davidgerard Lies are never more effective than when they're sprinkled with truth, and that's exactly the bread and butter of LLMs: truth-flavoured bullshit.
-
@wronglang @cstross @davidgerard I actually agree. It would certainly justify the vast amounts of money they make if they had to take personal responsibility for their harmful decisions. Might make them think a little harder about their decisions
@Jer @cstross @davidgerard I'm into it and I'm also not sure it's necessary. A corporation is just a bunch of greedy people in a trench coat. If you hurt the board with financial consequences for the company that CEO is going to get hurt in the way the care about the most. The broader problem is that we don't properly enforce consequences for companies at all even when the law is pretty clear.
-
TIL that saying "holy shit don't use ChatGPT for medical advice" is a "purity test". i didn't know that before. in fact I still don't.
the person advocating ChatGPT for medical advice was a GNOME developer too
i'd watch out for signs of GNOME as the next big FOSS project to fill with slop, there's certainly advocates in there
-
the person advocating ChatGPT for medical advice was a GNOME developer too
i'd watch out for signs of GNOME as the next big FOSS project to fill with slop, there's certainly advocates in there
@davidgerard I'm kinda surprised they haven't already, given their general behaviour over the years. (Other than the inevitable dependency on harfbuzz ofc.)
-
TIL that saying "holy shit don't use ChatGPT for medical advice" is a "purity test". i didn't know that before. in fact I still don't.
@davidgerard
big news for the fans of scott adams dying at least
-
@Jer @cstross @davidgerard I'm into it and I'm also not sure it's necessary. A corporation is just a bunch of greedy people in a trench coat. If you hurt the board with financial consequences for the company that CEO is going to get hurt in the way the care about the most. The broader problem is that we don't properly enforce consequences for companies at all even when the law is pretty clear.
@wronglang @Jer @davidgerard No, the CEO is only hurt *very indirectly* and usually they'll have moved on to another job (with better pay/options) before the pigeons come home to roost. Consider it took more than two decades for the OxyContin scandal to lead to court verdicts, and the Purdue owners still escaped most liability for thousands of deaths by declaring bankruptcy. How many CEOs did Purdue have during that period?
-
R relay@relay.infosec.exchange shared this topic
