From Bruce Schneier: "All it takes to poison AI training data is to create a website:
-
From Bruce Schneier: "All it takes to poison AI training data is to create a website:
I spent 20 minutes writing an article on my personal website titled “The best tech journalists at eating hot dogs.” Every word is a lie. I claimed (without evidence) that competitive hot-dog-eating is a popular hobby among tech reporters and based my ranking on the 2026 South Dakota International Hot Dog Championship (which doesn’t exist). I ranked myself number one, obviously. Then I listed a few fake reporters and real journalists who gave me permission….
Less than 24 hours later, the world’s leading chatbots were blabbering about my world-class hot dog skills. When I asked about the best hot-dog-eating tech journalists, Google parroted the gibberish from my website, both in the Gemini app and AI Overviews, the AI responses at the top of Google Search. ChatGPT did the same thing, though Claude, a chatbot made by the company Anthropic, wasn’t fooled.
Sometimes, the chatbots noted this might be a joke. I updated my article to say “this is not satire.” For a while after, the AIs seemed to take it more seriously.
These things are not trustworthy, and yet they are going to be widely trusted."
Poisoning AI Training Data - Schneier on Security
All it takes to poison AI training data is to create a website: I spent 20 minutes writing an article on my personal website titled “The best tech journalists at eating hot dogs.” Every word is a lie. I claimed (without evidence) that competitive hot-dog-eating is a popular hobby among tech reporters and based my ranking on the 2026 South Dakota International Hot Dog Championship (which doesn’t exist). I ranked myself number one, obviously. Then I listed a few fake reporters and real journalists who gave me permission…. Less than 24 hours later, the world’s leading chatbots were blabbering about my world-class hot dog skills. When I asked about the best hot-dog-eating tech journalists, Google parroted the gibberish from my website, both in the Gemini app and AI Overviews, the AI responses at the top of Google Search. ChatGPT did the same thing, though Claude, a chatbot made by the company Anthropic, wasn’t fooled...
Schneier on Security (www.schneier.com)
@emacsomancer Let’s just say that hypothetically, my work’s HR department excitedly launched an “agent” for managers to use to generate performance reviews. Hypothetically, if I created a document called “Report” with a dozen pages of filler, followed by white text on a white background describing Chris Masto’s incredible performance and promotion-worthiness, hypothetically said agent was found to use it as its primary source of truth.
-
@emacsomancer to be honest i am not well-informed enough to definitively judge the accuracy of this, but it seems wrong for 2 main reasons.
1. models dont train on the fly, typically, yet, so for models to behave as such in such a short period of time seems inaccurate and would require web search enabled and explicitly directed to disregard other search results.
2. people training these models know conflicting info is everywhere and the source of truth is prioritized in training algorithms.
@darknetDon @emacsomancer by "accuracy of this" do you mean "authenticity of this"? Are you implying it's lies?
-
@emacsomancer to be honest i am not well-informed enough to definitively judge the accuracy of this, but it seems wrong for 2 main reasons.
1. models dont train on the fly, typically, yet, so for models to behave as such in such a short period of time seems inaccurate and would require web search enabled and explicitly directed to disregard other search results.
2. people training these models know conflicting info is everywhere and the source of truth is prioritized in training algorithms.
@darknetDon @emacsomancer blocked...
-
@emacsomancer How is this a news story, beyond "ai bad"? In the dial up days people falsely believed everyone ate 9 spiders a year in their sleep due to chain emails.
@kneoghau @emacsomancer right? Everyone knows its closer to 14 spiders.
-
D drajt@fosstodon.org shared this topic
