Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because

[rndanger@infosec.exchange]

@korrupt @inthehands
Then my question is: Will Google claim that their AI search isn't subject to the old conventions and use that data to train AI and serve those results in their new format?

[theothersimo@mastodon.social]

@mjd @cceckman your bank account is available to malicious actors, but not available unconditionally. That’s a very big caveat to overlook.

[blindcoder@toot.berlin]

@inthehands I directly block on the webserver using https://perishablepress.com/ultimate-ai-block-list/ with a 403 Forbidden response.
I include anything containing "google" in the list.

[albinanigans@blackqueer.life]

@mjd @cceckman

I want people to access my content, not have it regurgitated in some slurry machine. I didn't sign up for that.

[markwyner@mas.to]

@inthehands crawlers choose whether or not they want to oblige robots.txt and meta noindex/nofollow.

The proper way to do this is add agent detection on the server-side, and force a 403. This essentially refuses a request.

This only works if you know all of the agents and they’re not using covert agents. Anyone can use any agent to crawl the web.

But the 403 solution is pretty solid overall.

[ticho@mas.to]

@khm @macronaut @hyc @inthehands I'm considering adding "agent" as one of the options for this regexp...

[praxeology@post.lurk.org]

@inthehands You could also set up a user-agent rule so that your web server gives the various google bots a tasty gas station sausage instead of the actual web page.

[shadsterling@mastodon.social]

@wronglang @mjd @cceckman right, which is distinct enough that it would be better to have a more distinct name for it

[jedbrown@hachyderm.io]

@inthehands It's important to note that search indexing is considered "transformative" and thus fair use *because* it does not supplant the market for the original content. That goes out the window when the product functions to capture traffic that would otherwise go to the cites. They are acting with impunity, but existing copyright law addresses this if courts find it to be not transformative.

[markwyner@mas.to]

@inthehands for a while I was hesitant to block Google. They have a psychological grip on us. We’re made to feel like we must play their game or our site doesn’t exist.

Fuck that. I’m out. I’m gonna block all of their bots. It’s gonna be 403 city.

[khm@hj.9fs.net]

yeah, the most comon one of those is meta-externalagent but that gets matched by wler because the url included has the word 'crawler' in it

CC: @macronaut@mas.to @hyc@mastodon.social @inthehands@hachyderm.io

[ferrix@mastodon.online]

@inthehands @adamshostack it's transformative which makes it a very uncertain fight

[gudenau@hachyderm.io]

@inthehands Good point, I ought to setup my stuff to serve robots.txt...

[weizenspreu@chaos.social]

@inthehands This won‘t necessarily help prevent the ingestion into LLMs. Have a look at the TDM Reservation Protocol.

[coho@climatejustice.social]

@inthehands
There is a new fad called "data poisoning" that web sites are using to foil ai scraping. One music site put a Homer Simpson monologue into every track in its online data base. It starts a few seconds in and continues to the end. That's only one way it's being used. We need a generation of ai "monkey wrench gangs " to start sabotaging. It's really no different than what Edward Abbey talked about, instead of extractive earth raping machinery being targeted , it's data mining machinery.

[nicelymanifest@mastodon.social]

@inthehands The days of 'do no evil' long forgotten. Money as well as power can deeply corrupt.

[pionir@masto.bike]

@inthehands I came across a bit bomb for AI agents but I can't remember the name (sorry!). It worked by making a small zip payload extract to a massive file to overload the agent / crawlers. Not sure how well it works though. I'll see if I can find it.

[albertcardona@mathstodon.xyz]

@schamschula @inthehands

Mind sharing the necessary subset of the nginx config to enforce robots.txt as an nginx block list? Thank you.

[accordionbruce@mastodon.social]

@markwyner @inthehands
There is a point where their search becomes bad enough that being on Google search has less and less payoff

[jima@mspsocial.net]

@khm @inthehands I'm more interested in sending it to AI scrapers, not that they tend to identify themselves honestly.