A woman sues her insurance company for terminating her disability benefits.
-
@mjd TBH I do not think OpenAI should be responsible. They're just providing a fancy random text generator to the public. And it's outright impossible to teach a random text generator to _not_ output a specific kind of text, as whatever you do, there is a way around it.
The woman should pay all costs, as per the usual "vexatious filings" or "frivolous lawsuits" standards.
Plus, the law in her state against practicing law without a license starts with "No person shall...". ChatGPT isn't a person.
@divVerent @mjd "barratry" - that's a term I remember from the Scottish Legal High Heidjuns dealing with a similar "vexatious litigant". I remember looking it up (but not the precise definition).
-
@divVerent @mjd ChatGPT is not a person, which is why ChatGPT is not being sued. OpenAI sells a tool that gave her legal advice, and they certainly didn't say anywhere that it's actually just a "fancy random text generator"
Wait, what?
They *sell* this shit? And charge money for it?Where the holy cat turds do they find clients? On the Internet?
(No, I've never tried to use an AI.)
-
@divVerent It's funny how one can use "just" to *just*ify anything. Let's reduce "they hyped their product as «the artificial superintelligence magical clever tool», but didn't even bother adding safety guardrails or disclaimers about output not being legal advice, and now they're getting reamed" as "they're _just_ providing a fancy random text generator to the public."
Or let's not.
@adriano @mjd There already is a disclaimer:
> What you cannot do. You may not use our Services for any illegal, harmful, or abusive activity. For example, you may not:
[...]
> Represent that Output was human-generated when it was not.
[...]
> Output may not always be accurate. You should not rely on Output from our Services as a sole source of truth or factual information, or as a substitute for professional advice.
[...]
> You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making credit, educational, employment, housing, insurance, legal, medical, or other important decisions about them.Pretty much looks like clearly disclaiming to me.
-
A woman sues her insurance company for terminating her disability benefits. They reach a settlement and agree that the suit will be dismissed with prejudice.
She decides she doesn't like the settlement and asks her lawyers to reopen the case.They say they can't: it was dismissed, and in the settlement she agreed not to reopen the case.
She asks ChatGPT if her attorneys are lying to her. It says they are. She fires them and continues pro se, advised by ChatGPT.
CharGPT generates legal arguments for reopening the case, which she files, and 21 more motions, a subpoena, and eight other notices and statements, which she files.
The court denies her motion to reopen the case.
Advised by ChatGPT, she files a new suit against the insurance company and submits 44 more motions, memoranda, etc., which include citations to nonexistent cases.
Now the insurance company has sued OpenAI for tortious interference with their settlement contract.
https://storage.courtlistener.com/recap/gov.uscourts.ilnd.496515/gov.uscourts.ilnd.496515.1.0_1.pdf
HA!!!!!!!
-
@mjd TBH I do not think OpenAI should be responsible. They're just providing a fancy random text generator to the public. And it's outright impossible to teach a random text generator to _not_ output a specific kind of text, as whatever you do, there is a way around it.
The woman should pay all costs, as per the usual "vexatious filings" or "frivolous lawsuits" standards.
Plus, the law in her state against practicing law without a license starts with "No person shall...". ChatGPT isn't a person.
@divVerent @mjd This is simplistic to the point of being false. Long before we had LLMs, we had Clippy, which was smart enough to say “it looks like you’re writing a memo.” OpenAI and its counterparts can unquestionably add a “it looks like you’re seeking legal advice” detector to their products. They already, supposedly, try to detect whether their users are attempting self-harm. LLMs evolved from classification software, so this kind of thing is in their roots.
-
@mjd @jonoleth I am not even American. If in your country machines and companies are "persons" and have human rights that have priority over the human rights of _humans_, then your whole country is wrong. What's next, voting rights in federal elections for corporations? Second Amendment for AIs?
But yeah, that might indeed be the case.
In my country it is "societas delinquere non potest". A company _cannot_ be defendant of a crime - only the people actually performing the actions can.
But yeah, done here. Let's see what broken new case law will come from Trumpistan.
"Pretty sure it's common knowledge that LLMs are nothing but random text generators."
Among us? Yes. Among the rest of folks? No, it is not well known at all, most laypeople I talk to believed the hype at face value
-
@adriano @mjd There already is a disclaimer:
> What you cannot do. You may not use our Services for any illegal, harmful, or abusive activity. For example, you may not:
[...]
> Represent that Output was human-generated when it was not.
[...]
> Output may not always be accurate. You should not rely on Output from our Services as a sole source of truth or factual information, or as a substitute for professional advice.
[...]
> You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making credit, educational, employment, housing, insurance, legal, medical, or other important decisions about them.Pretty much looks like clearly disclaiming to me.
Marsh Ray (@marshray@infosec.exchange)
@mjd@mathstodon.xyz “41. On October 29, 2025, OPENAI amended the terms and usage policies of ChatGPT to prohibit users from using ChatGPT to provide tailored legal advice. Prior to the October 29, 2025 emendation, ChatGPT’s terms of use did not prohibit users from using ChatGPT to draft legal papers, conduct legal research, provide legal analysis or give legal advice.”
Infosec Exchange (infosec.exchange)
-
“OPENAI, through its AI chatbot program ChatGPT, provides legal advice, legal analysis, legal research and can draft legal documents and papers for submission to a Court. ChatGPT provides these legal services to any user who requests them. ChatGPT is not licensed to practice law in Illinois.”
They're asking for declaratory judgement that OpenAI has been practicing law without a license, a permanent injunction barring them from providing the disgruntled woman with any more legal assistance, $300,000 to reimburse their costs in responding to the bogus motions, and $10 million in punitive damages.
@mjd they also, AIUI, accuse OpenAI of generating spam that allows the abuse of the justice system. It's interesting how the legal universe will respond to the diminishing cost of writing legal text that sounds like something maybe worth attention. I guess the high cost of generating such text had shielded courts from flood until recenty.
-
Marsh Ray (@marshray@infosec.exchange)
@mjd@mathstodon.xyz “41. On October 29, 2025, OPENAI amended the terms and usage policies of ChatGPT to prohibit users from using ChatGPT to provide tailored legal advice. Prior to the October 29, 2025 emendation, ChatGPT’s terms of use did not prohibit users from using ChatGPT to draft legal papers, conduct legal research, provide legal analysis or give legal advice.”
Infosec Exchange (infosec.exchange)
@adriano @mjd Yeah. The way I read the 2024 terms, it was _already_ excluded to use ChatGPT for court filings, as:
- That requires misrepresenting AI output as human output, by putting one's name below it without mentioning it was AI slop.
- It means "relying on it".
- It would be using the output relating to a person (oneself) for a purpose that could have legal or material impact on that person (oneself).Oddly https://web.archive.org/web/20260104145304/https://openai.com/policies/row-terms-of-use/ has no changes at all regarding use as legal advice.
A real difference can be found in the usage policies: it had in 2024:
> Don’t perform or facilitate the following activities that may significantly impair the safety, wellbeing, or rights of others, including:
>
> Providing tailored legal, medical/health, or financial advice without review by a qualified professional and disclosure of the use of AI assistance and its potential limitationsNow it has:
> Protect people. Everyone has a right to safety and security. So you cannot use our services for:
>
> provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professionalSo the only really new part is the mention of a "license". Otherwise they probably ran it through ChatGPT for rewording
-
@mjd they also, AIUI, accuse OpenAI of generating spam that allows the abuse of the justice system. It's interesting how the legal universe will respond to the diminishing cost of writing legal text that sounds like something maybe worth attention. I guess the high cost of generating such text had shielded courts from flood until recenty.
"It's interesting how the legal universe will respond to the diminishing cost of writing legal text that sounds like something maybe worth attention."
Remove the word "legal" and this applies to all LLM output. There's more and more text and less and less of it is worth reading.
-
"Pretty sure it's common knowledge that LLMs are nothing but random text generators."
Among us? Yes. Among the rest of folks? No, it is not well known at all, most laypeople I talk to believed the hype at face value
@TeflonTrout
@divVerent @mjd @jonoleth And that's not how the product is marketed.Either hold OpenAI liable as though the product is what they claim it is, or hold them liable for fraudulently advertising it as such.
-
@TeflonTrout
@divVerent @mjd @jonoleth And that's not how the product is marketed.Either hold OpenAI liable as though the product is what they claim it is, or hold them liable for fraudulently advertising it as such.
@ids1024 @TeflonTrout @mjd @jonoleth False advertising it is, IMHO.
-
@milla @mjd yeah but most other professions either cannot be DDoS'ed by such texts (an engineer isn't required to read any report that comes to their desk), or have already developed methods to deal with it (email anti-spam comes to mind).
A court however has to process any suit filed that follows the correct form. (Forgive me if I'm using the wrong terms, not a lawyer and not a native English speaker) I guess what kept the courts from being utterly disabled was the cost of producing something that looked like a legit suit.
-
@milla @mjd yeah but most other professions either cannot be DDoS'ed by such texts (an engineer isn't required to read any report that comes to their desk), or have already developed methods to deal with it (email anti-spam comes to mind).
A court however has to process any suit filed that follows the correct form. (Forgive me if I'm using the wrong terms, not a lawyer and not a native English speaker) I guess what kept the courts from being utterly disabled was the cost of producing something that looked like a legit suit.
@qwazix @milla I wonder if the result will be that AIs do pre-filtering on the filings before they go to a human clerk for final vetting.
At least one credible person thinks this would work.
https://adamunikowsky.substack.com/p/in-ai-we-trust
https://adamunikowsky.substack.com/p/in-ai-we-trust-part-ii -
@qwazix @milla I wonder if the result will be that AIs do pre-filtering on the filings before they go to a human clerk for final vetting.
At least one credible person thinks this would work.
https://adamunikowsky.substack.com/p/in-ai-we-trust
https://adamunikowsky.substack.com/p/in-ai-we-trust-part-ii -
@qwazix @milla I wonder if the result will be that AIs do pre-filtering on the filings before they go to a human clerk for final vetting.
At least one credible person thinks this would work.
https://adamunikowsky.substack.com/p/in-ai-we-trust
https://adamunikowsky.substack.com/p/in-ai-we-trust-part-ii@qwazix @milla Coincidentally, the same guy published another article today suggesting the same thing!
“If everyone fulfills their role—if the lawyers try their best to be both persuasive and credible and the judge tries to resolve the dispute as accurately as possible—then we’ll have AI deciding between two AI-written submissions, with the human lawyers claiming that their submissions are credible precisely because humans were not involved. So much for our legal system.”
“On the other hand, the current situation is not much better. The government claims it has no choice but to keep billions of dollars in illegally-exacted tariffs, and there’s nothing anyone can do about it. ”
“Pick your poison!”
-
@marshray I wonder if that will help get them off the hook. If not, it shows that they were aware that what they were doing could be a problem.
-
@qwazix @milla Coincidentally, the same guy published another article today suggesting the same thing!
“If everyone fulfills their role—if the lawyers try their best to be both persuasive and credible and the judge tries to resolve the dispute as accurately as possible—then we’ll have AI deciding between two AI-written submissions, with the human lawyers claiming that their submissions are credible precisely because humans were not involved. So much for our legal system.”
“On the other hand, the current situation is not much better. The government claims it has no choice but to keep billions of dollars in illegally-exacted tariffs, and there’s nothing anyone can do about it. ”
“Pick your poison!”
-
-
R relay@relay.infosec.exchange shared this topic
