Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands This won‘t necessarily help prevent the ingestion into LLMs. Have a look at the TDM Reservation Protocol.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands
There is a new fad called "data poisoning" that web sites are using to foil ai scraping. One music site put a Homer Simpson monologue into every track in its online data base. It starts a few seconds in and continues to the end. That's only one way it's being used. We need a generation of ai "monkey wrench gangs " to start sabotaging. It's really no different than what Edward Abbey talked about, instead of extractive earth raping machinery being targeted , it's data mining machinery. -
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands The days of 'do no evil' long forgotten. Money as well as power can deeply corrupt.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands I came across a bit bomb for AI agents but I can't remember the name (sorry!). It worked by making a small zip payload extract to a massive file to overload the agent / crawlers. Not sure how well it works though. I'll see if I can find it.
-
R relay@relay.infosec.exchange shared this topic
-
@inthehands If they ignore robots.txt, they will be added to the block list in nginx.conf. My robots.txt has a note stating as much. There is plenty of company there!
Mind sharing the necessary subset of the nginx config to enforce robots.txt as an nginx block list? Thank you.
-
@inthehands for a while I was hesitant to block Google. They have a psychological grip on us. We’re made to feel like we must play their game or our site doesn’t exist.
Fuck that. I’m out. I’m gonna block all of their bots. It’s gonna be 403 city.
@markwyner @inthehands
There is a point where their search becomes bad enough that being on Google search has less and less payoff -
@khm @inthehands I'm more interested in sending it to AI scrapers, not that they tend to identify themselves honestly.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands I am all in on reciprocating... what do we do?
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands Two quotes from Pratchett comes to mind
>>> “All witches are selfish, the Queen had said. But Tiffany’s Third Thoughts said: Then turn selfishness into a weapon! Make all things yours! Make other lives and dreams and hopes yours! Protect them! Save them! Bring them into the sheepfold! Walk the gale for them! Keep away the wolf! My dreams! My brother! My family! My land! My world! How dare you try to take these things, because they are mine!
>>> "We look to ... the edges," said Mistress Weatherwax. "There's a lot of edges, more than people know. Between life and death, this world and the next, night and day, right and wrong ... an' they need watchin'. We watch 'em, we guard the sum of things. And we never ask for any reward. That's important.”
-
@korrupt @inthehands
Then my question is: Will Google claim that their AI search isn't subject to the old conventions and use that data to train AI and serve those results in their new format?@RnDanger @inthehands well, we don’t know and we will see. My guess are separate scrapers (officially) and a lot of mistrust (are there others?) and masses of unidentified scrapers. Nevertheless, Google can better afford to play by the rules, since hey already own the largest index. Think also of Video etc. Will volume win the war? Or quality and freshness? Etc. Future is difficult.
-
Mind sharing the necessary subset of the nginx config to enforce robots.txt as an nginx block list? Thank you.
@albertcardona @inthehands It involves a couple steps, given the idiosyncrasies of the nginx regex support (no full pcre here!).
I keep two classes of blocked agents: (1) bad agents; and (2) scrapping false agents. A third regex unblocks agents that are false positives (due to (2)). -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands If I understand your question correctly (sorry if it's not the case) I think that Anubis, the AI crawler protection, could be part of the solution. Not only would that work for Google, that would (or at least *should*) also work against other crawlers.
Another advantage is that it can work along your other solutions.
OTHO the drawback is that it would work against all crawler, so you would "disappear" from every search engine...
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands thank you friend. Adding your quote to my common place book
-
@inthehands this is a fence-post defense against this, google Will Not Care
just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output
@ShadowJonathan @inthehands they alsof are pretty involved with the contents of the standard (75% of the authors), so luring the crawler into a pit of crappy data is probably your only way to protest besides avoiding to hand any of your money and attention to them.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands I fear that when we block the bot, they'll access the site through Chrome. Getting what people browsing get. I don't know what could stop them.
-
@inthehands I know of at least one professional artist who has deliberately poisoned their images, in an attempt to deter AI scraping (mostly because the scrapers blast her small site and effectively DoS it). If they follow robots.txt, they're not affected... but they were already ignoring robots.txt
I just read an IARPA paper that said poisoning as little as .1% of training data can disrupt a model. If content creators choose to deliberately poison content that they ask not to be scraped, it might be a nice way to deter bad behavior.
The tools I know of work on imagery, but with effort people may come up with stuff that works on data as well. E.g., burying base64-encoded malicious prompts in your text, posting tables as poisoned images rather than text, etc.
Seems like we should start organizing and taking firm action now, before AI companies start buying politicians and making such defenses illegal.
@mathaetaes @inthehands > posting tables as poisoned images rather than text
Please **never** do that. Accessibility is more important than poisoning LLMs.
-
@macronaut @khm @inthehands the update would change that 403 to a 402. And add "error_page 402 /402.html;" to the server{} config, and create the /402.html file in the docroot containing whatever desired message.
@macronaut @khm @inthehands maybe an appropriate message would be "send 1XMR to be unblocked"...
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands @philbaker1 I don't think cynicism is surrender. Cynicism is being aware of what could happen and preparing yourself mentally for it. Just because I'm cynical about some things doesn't mean I've given into them. It means I'm aware of them and I know how bad they can get.
-
@mathaetaes @inthehands > posting tables as poisoned images rather than text
Please **never** do that. Accessibility is more important than poisoning LLMs.
@raulmatias @inthehands Ooh, good point - I had completely forgotten about screen readers in that context.
