Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit?
-
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender Really helpful. Thanks for sharing.
I often reason about it this way. There are very few things like this where, if you opt out this time, you can’t opt in next time. On the other hand, there are LOTS of situations, this being a good example, where opting out after you opted in is substantially more effort (or even impossible).
Opting out by default is usually a safe thing to do. You can always opt in later if you change your mind.
-
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender multiple times in fact. And I hate it.
-
Yep. And I said no. She initially said not to worry because it's all deleted afterwards. I said that, no, it is not. That's not how LLM's work. All that data remains in there somewhere and can be hacked, plus I don't want anything about me used to train those things on principal. She didn't argue.
@Mikal @emilymbender
It's pinky promised by LLM vendors to the healthIT vendors selling to hospitals that data sent to their servers for the purpose of LLM input is not retained or used outside of inference.HealthIT vendors are then happy to believe this at face value so they can sell AI hype to hospital execs desperate to force their clinicians to shove more patients through at any cost and slash labor costs.
Docs are told it's fine by IT who was told it's fine by the CIO who was told...
-
@emilymbender I do share a lot of AI skepticism, but physician perspective (I use it about 25-30% of visits), there are many highly speculative aspects of this take:
🧵 1/5
1) Point #1 is valid, however, the same data safety questions can be asked regarding other integrated systems. Like where is your EMR data stored, how does your radiology data integrate (reviewed in 3rd party software), etc.
2) Consent: valid concern, but the fullest version would be a long EULA-like text with a checkbox...@P__X Your experience is your experience, but I am **appalled** at what you're saying about consent here. The fullest version would be too long, so we're not actually doing informed consent? No thank you.
-
@Mikal @emilymbender
It's pinky promised by LLM vendors to the healthIT vendors selling to hospitals that data sent to their servers for the purpose of LLM input is not retained or used outside of inference.HealthIT vendors are then happy to believe this at face value so they can sell AI hype to hospital execs desperate to force their clinicians to shove more patients through at any cost and slash labor costs.
Docs are told it's fine by IT who was told it's fine by the CIO who was told...
@Mikal @emilymbender
Just to add a little more here: I listened to the R&D head of a large allegedly monopolistic EHR vendor tell a room full (1000s) of physicians and hospital CXOs they need to be less cautious and go full speed ahead in their adoption of AI.It's being pushed really, really hard in healthcare software.
-
@meltedcheese @emilymbender It’s very likely this feature was introduced into the medical office patient management software. It’s likely being pushed hard by the developers. It might even offer a kickback scenario for the doctors who record the most. Doctors are not going to argue with free money, but they will argue with patients if they stand to lose that kickback money.
This suggests a deeper journalistic dive into that patient mgmt. software might be justified.
@randocity @meltedcheese @emilymbender my medical defence insurer encourages its use because evidence shows lower litigation rates in consultations with scribes. The study was done using human scribes, predates the AI era.
-
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender for a doctors perspective on the more profound side effects of “efficiency”
I Was an Enthusiastic Early Adopter of AI Scribes. Here’s Why I Stopped
A GP reflects on what eighteen months of ambient scribing taught them about the consultation they thought they already understood.
(benngooch.substack.com)
“I felt myself becoming a passive observer in encounters where I had previously been an active architect. I felt my clinical memory, my narrative identity, and my sense of connection to my patients beginning to erode at the edges.”
-
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender psychiatry did it without informed consent. I am livid
-
@randocity @jrdepriest @emilymbender
I think that kind of depends on things like state laws. California for example is a two party consent state so I think recording someone without asking might actually be a criminal offense. Plus they have to have some sort of device and that is likely to be visible. Either way, I think that's why we need to push back immediately and make sure they understand that this is not acceptable.
@Mikal @jrdepriest @emilymbender The problem is even knowing the doctor’s office recorded the interaction, other than via a whistleblower. If the recording is transcribed and then discarded by the doctor, how would a patient ever know? Once doctors realize they basically can’t get caught doing it, what or who will stop them?
Insurance companies aren’t going to care or even ask if the doctors collected their recordings illegally.
️ -
@randocity @meltedcheese @emilymbender my medical defence insurer encourages its use because evidence shows lower litigation rates in consultations with scribes. The study was done using human scribes, predates the AI era.
@M3L155A @meltedcheese @emilymbender AI hasn’t yet proven itself to be a reliable or useful witness in legal cases. I’m not even sure a lawyer has yet tried using an AI as a witness.
It is possible that having audio recordings of patient interactions could prove useful in courts, but that implies that doctors are being sued more now than in the past.
Insurers don’t derive benefits from AI directly, so I don’t understand this push.
-
@M3L155A @meltedcheese @emilymbender AI hasn’t yet proven itself to be a reliable or useful witness in legal cases. I’m not even sure a lawyer has yet tried using an AI as a witness.
It is possible that having audio recordings of patient interactions could prove useful in courts, but that implies that doctors are being sued more now than in the past.
Insurers don’t derive benefits from AI directly, so I don’t understand this push.
@M3L155A @meltedcheese @emilymbender For clarification, when I say that insurers aren’t deriving benefits from AI directly, I mean specifically the AI that’s being used in doctors offices, learning from patient recordings.
It is very possible, however, that insurance companies are using AI in their own internal systems, but those AI systems are entirely separate from AI used in doctor’s office patient systems.
-
@Mikal @jrdepriest @emilymbender The problem is even knowing the doctor’s office recorded the interaction, other than via a whistleblower. If the recording is transcribed and then discarded by the doctor, how would a patient ever know? Once doctors realize they basically can’t get caught doing it, what or who will stop them?
Insurance companies aren’t going to care or even ask if the doctors collected their recordings illegally.
️@Mikal @jrdepriest @emilymbender The only place where a doctor might get caught at doing it is if they produce a recording in a court of law as part of a legal case. The problem is, patient recordings would be considered part of HIPAA compliance and may be inadmissible if proper procedures are not followed.
There are definitely procedures to follow when introducing HIPAA data into a court case, such as giving the patient a chance to object.
-
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender Funnily enough, transcribing can preserve privacy without issues. Whisper.cpp runs decently well on phones, and can be run on servers that process patient records under the same security constraints. Could easily be run locally even.
Problem is, that’s extremely hard to prove in the current „just slap a gear on it and call it steampunk” climate. I would definitely not trust a random provider.
And if they do „summarization”, forget privacy.
-
@P__X Your experience is your experience, but I am **appalled** at what you're saying about consent here. The fullest version would be too long, so we're not actually doing informed consent? No thank you.
@emilymbender Agreed. For any points that were valid, none of them necessitate the use of LLMs. Never mind without consent. Disgusting.
-
@commonst @emilymbender Medical providers are one to point fingers at patients for being tech naïve. Medical providers, and the medical industry in general, are notoriously the worst at being informed about tech; worse than any industry short of lawyers. That’s actually why HIPAA exists.
@randocity @emilymbender I am in zcanada. No HIPAa, but we do tend to go where the US goes on a lot of things.
-
@P__X Your experience is your experience, but I am **appalled** at what you're saying about consent here. The fullest version would be too long, so we're not actually doing informed consent? No thank you.
@emilymbender "he fullest version would be too long, so we're not actually doing informed consent?"
No, that is not what is being said there. Unlike a blog post, I am restricted in space. I explicitly said that is is a valid concern. A basic research consent form is 8+ pages of legalese and I'm afraid that the future solution will be to add it as a checkbox for 30 pages of text at check-in that nobody reads and doesn't actually inform better. And again, my point #1.
-
@emilymbender Agreed. For any points that were valid, none of them necessitate the use of LLMs. Never mind without consent. Disgusting.
1) Consent is always obtained (and documented). The ideal way/length/detail to do it is up for debate. A 30 page EULA (if this is outsourced to the legal department) will not provide better informed consent, however. The Sutter lawsuit might propel better regulation and policies.
2) Nothing necessitates the use of LLMs. It doesn't mean that it can't be helpful in certain use cases, which I spent my time to point out hoping for a convo and not selective dismissal.
-
@netopwibby Oof -- so she asked if you were okay being recorded but did not provide info on what was going to happen to the recording?
@emilymbender Had a similar experience to
@netopwibby one with a cardiologist, I am in Canada. But at the last appointment she didn’t seem to use it? Will try to think to ask her about the use of it next time if I don’t forget. -
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender
I have-- and refused! -
Have you been asked by a medical provider recently for consent to have an "AI" scribe record your visit? Us, too. And we have **thoughts**
Why you should refuse to let your doctor record you
By: Emily M. Bender and Decca Muldowney At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started,...
(buttondown.com)
@emilymbender no but in the agreement they ask us to sign periodically it said that they might use AI. So I said I wasn’t signing if they were going to. They asked the doc and she said no I don’t use AI transcription at all and I didn’t know that was in there!
