Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because

[datarama@hachyderm.io]

@inthehands On the one hand, I agree - if nothing else, because caring, doing and building are what I *want* to do.

But on the other hand: "We will rebel against the AI oligarchs by creating *even better* training data for them!"

[dalias@hachyderm.io]

@khm @inthehands I wonder if this has legal implications for their bypassing it...

[tk@f.kawa-kun.com]

@inthehands @khm Just did that after seeing your post.

[mjd@mathstodon.xyz]

@theothersimo @cceckman I didn't suggest “should be”. I will stand by “All information on the WWW is available to malicious actors”.

You seem surprised at this fact, and say the premise is “idiotic”. Okay.

[elithebearded@fed.qaz.red]

@inthehands

Feeling better about having added Googlebot to robots.txt last year

[sollat@masto.ai]

@inthehands
How long until Maps will only give directions in real-time, no destination foreknowledge allowed.

[ai6yr@m.ai6yr.org]

@inthehands I already had blocked them off my Mastodon server (albeit, maybe not fully successful, seems like it still shows links).

[rndanger@infosec.exchange]

@korrupt @inthehands
Then my question is: Will Google claim that their AI search isn't subject to the old conventions and use that data to train AI and serve those results in their new format?

[theothersimo@mastodon.social]

@mjd @cceckman your bank account is available to malicious actors, but not available unconditionally. That’s a very big caveat to overlook.

[blindcoder@toot.berlin]

@inthehands I directly block on the webserver using https://perishablepress.com/ultimate-ai-block-list/ with a 403 Forbidden response.
I include anything containing "google" in the list.

[albinanigans@blackqueer.life]

@mjd @cceckman

I want people to access my content, not have it regurgitated in some slurry machine. I didn't sign up for that.

[markwyner@mas.to]

@inthehands crawlers choose whether or not they want to oblige robots.txt and meta noindex/nofollow.

The proper way to do this is add agent detection on the server-side, and force a 403. This essentially refuses a request.

This only works if you know all of the agents and they’re not using covert agents. Anyone can use any agent to crawl the web.

But the 403 solution is pretty solid overall.

[ticho@mas.to]

@khm @macronaut @hyc @inthehands I'm considering adding "agent" as one of the options for this regexp...

[praxeology@post.lurk.org]

@inthehands You could also set up a user-agent rule so that your web server gives the various google bots a tasty gas station sausage instead of the actual web page.

[shadsterling@mastodon.social]

@wronglang @mjd @cceckman right, which is distinct enough that it would be better to have a more distinct name for it

[jedbrown@hachyderm.io]

@inthehands It's important to note that search indexing is considered "transformative" and thus fair use *because* it does not supplant the market for the original content. That goes out the window when the product functions to capture traffic that would otherwise go to the cites. They are acting with impunity, but existing copyright law addresses this if courts find it to be not transformative.

[markwyner@mas.to]

@inthehands for a while I was hesitant to block Google. They have a psychological grip on us. We’re made to feel like we must play their game or our site doesn’t exist.

Fuck that. I’m out. I’m gonna block all of their bots. It’s gonna be 403 city.

[khm@hj.9fs.net]

yeah, the most comon one of those is meta-externalagent but that gets matched by wler because the url included has the word 'crawler' in it

CC: @macronaut@mas.to @hyc@mastodon.social @inthehands@hachyderm.io

[ferrix@mastodon.online]

@inthehands @adamshostack it's transformative which makes it a very uncertain fight

[gudenau@hachyderm.io]

@inthehands Good point, I ought to setup my stuff to serve robots.txt...