I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y.
-
a Google spokesperson said such results reflected the performance of an earlier model. They added, “We have always been transparent about the limitations of generative AI and provide in-app prompts to encourage users to double-check information. For sensitive matters such as medical advice, Gemini recommends users consult with qualified professionals.”
that would mostly work well if they released 'AI overview' as an opt-in feature instead of forcing it on users who have some trust in Google built over the past decades and don't expect it to suddenly start making stuff up lol
@mkljczk @ElenLeFoll
Good point.Google, if Gemini is as useful as you hope it will be, it is inevitable that it will just come to be known as "Google" and AI answers to direct questions is just a feature of Google search.
Google, if Gemini is unreliable and can not be reliable, why are you letting it tarnish your brand?
-
@ElenLeFoll Yeah that's totally getting ingested into my Zotero along with all the other corroborating evidence for "The only way is down". But to falsify I am looking at what working code generation studies there actually are.
Johnathan Swift described The Machine (for writing) in 1726; https://en.wikipedia.org/wiki/The_Engine
-
@bencourtice @FediThing @ElenLeFoll
We have a whole module on how to cite and also how AI works (ethics, errors, foundational knowledge, all that) for our freshman public health students. We STILL had over 10% academic affairs referrals for using hidden prompts and hallucinated citations and that's with a requirement that they give us annotated PDF copies of every cited paper.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
boost with CN: "AI"
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
The obvious end goal of AI is centralized control of information that can be used to bend public opinion, win elections for pedophiles, criminals and set trends.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
@ElenLeFoll LLMs, like seagulls, swallow anything that's thrown at them. It's known as 'gullibility' for a good reason.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
@ElenLeFoll I'm sorry, they put up fake preprints and then said other researchers citing these preprints are the problem? Standing up a fake preprint is absurdly unethical
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
This is a rerun of the Sokal Hoax.
Like some here, some folks thought the Sokal hoax was ethically problematic, but the postmodernists needed a wake up call, as do the LLM fans.
Really: the LLM idea is the stupidest* thing to come out of Computer Science ever. We need to be embarrassed.
*: Unnecessary explanation: the idea that random text generation has something to do with intelligence is really really stupid.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
That's something the government does every few years already. Nothing new.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
@ElenLeFoll I'm convinced one of the main reasons we die is that we get too old or too sick to manage our own healthcare. We can't do the research to find the right studies, we can't read or understand those studies, and we can't question our providers to be sure they actually understand what's wrong with us.
Once we are dependent on mere employees, the quality of care goes way down, and mistakes get made, or our treatment is just ineffective.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
as a scientist, you are surely aware of bias
such as the bias on social media, where if AI does something bad, it gets widely disseminated (goes viral)
whereas if AI does something good, no one talks about it
also,
PSA
when I was a baby PhD student in 1985, my teachers warned me over and over, don't trust something just cause it is published in a peer reviewed journal
be careful of all that you read !!!!! -
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
@ElenLeFoll You might enjoy my recent experience with raclette maximalism.
-
@ElenLeFoll I love this quote about the original article. “Acknowledgments also thanked “Professor Sideshow Bob” and a professor from the Starfleet Academy for access to a lab aboard the USS Enterprise”
#AI #AiSlop@Pineywoozle Yes, big cinema !
@ElenLeFoll -
Johnathan Swift described The Machine (for writing) in 1726; https://en.wikipedia.org/wiki/The_Engine
@mycotropic @bms48 @ElenLeFoll shout-out to Ramon Llull: http://www.computer-timeline.com/timeline/ramon-llull/
-
@ElenLeFoll Good. Poison as much as possible, let it eat its own crap and dance while it burns to the ground.
@MeneerDeBruin As a professional 100% human writer, I'm indeed interested in how we could use our creativity for that goal!
@ElenLeFoll -
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
@ElenLeFoll i *like it a lot* when people teach AI... wrong stuff. Piss in the well at every opportunity, folks.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
It seems like the real underlying problem is the "publish or perish" syndrome, where the value of a researcher is based on how many papers they write, or how often they're reference.
So there's a proliferation of papers, many of which are meaningless, and which no one has time to actually read, being referenced in other papers by other researchers who don't have time to read and evaluate all these other papers.
-
I was only made aware of this (frankly awesome) case of LLM poisoning today: https://www.nature.com/articles/d41586-026-01100-y. A researcher made up a disease and published two evidently fake preprints about it (including sentences such as “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”), which were almost immediately picked up by LLMs and documented in their output. Worse, actual – supposedly serious – medical papers also started citing the preprints, demonstrating that academics relying on LLMs to do their work is a genuine problem! Not that I had my doubts but, if anyone did, this seems like the perfect demonstration of the problem. Article immediately added to the syllabus of the class I am co-teaching with Iris Ferrazzo on LLMs for Romance Studies/Humanities!
#LLM #GenAI #academia #research #ResearchIntegrity #humanities
@ElenLeFoll The whole thing is like an oil spill.
-
@mycotropic @bms48 @ElenLeFoll shout-out to Ramon Llull: http://www.computer-timeline.com/timeline/ramon-llull/
That's an amazing history and I'll include llull in my "History of AI/LLM" lecture! Thanks!
-
@ElenLeFoll I love this quote about the original article. “Acknowledgments also thanked “Professor Sideshow Bob” and a professor from the Starfleet Academy for access to a lab aboard the USS Enterprise”
#AI #AiSlop@Pineywoozle @ElenLeFoll what a fun piece of work really, to write made-up scientific articles with all the silly things you want!
