<![CDATA[Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because]]>RE: https://tldr.nettime.org/@tante/116605858023186072

Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because

and •only because•

they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.

Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.

1/2

]]>https://board.circlewithadot.net/topic/d8555f73-2ef0-4a8a-adea-b62fe23ac444/google-search-rests-on-a-social-contract-their-bots-can-crawl-our-sites-they-can-index-our-sites-and-they-can-show-excerpts-of-our-sites-becauseRSS for NodeMon, 25 May 2026 13:22:31 GMTWed, 20 May 2026 14:49:24 GMT60<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:33:05 GMT]]>@blogdiva

Keep it in pencil. I’m still learning myself, and not sure I understand everything correctly here.

]]>https://board.circlewithadot.net/post/https://hachyderm.io/users/inthehands/statuses/116608806794415503https://board.circlewithadot.net/post/https://hachyderm.io/users/inthehands/statuses/116608806794415503Wed, 20 May 2026 20:33:05 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:26:50 GMT]]>@inthehands TIL thanks

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/blogdiva/statuses/116608782184580486https://board.circlewithadot.net/post/https://mastodon.social/users/blogdiva/statuses/116608782184580486Wed, 20 May 2026 20:26:50 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:22:46 GMT]]>@blogdiva
I believe that my various name=“___” values specifically target Google.

Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)

]]>https://board.circlewithadot.net/post/https://hachyderm.io/users/inthehands/statuses/116608766215062266https://board.circlewithadot.net/post/https://hachyderm.io/users/inthehands/statuses/116608766215062266Wed, 20 May 2026 20:22:46 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:15:50 GMT]]>instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?

there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.

@inthehands

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/blogdiva/statuses/116608738917100702https://board.circlewithadot.net/post/https://mastodon.social/users/blogdiva/statuses/116608738917100702Wed, 20 May 2026 20:15:50 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:15:00 GMT]]>@raulmatias @inthehands @joe @ShadSterling There's the built-in one, and another - slightly more complex - in Nam-Shub of Enki.

There will be more templates coming in the next few months (and new scripts!).

But a lot of things are doable today, if someone takes the time and creates a suitable template.

]]>https://board.circlewithadot.net/post/https://come-from.mad-scientist.club/users/algernon/statuses/01KS3GDYJWX7S92CP9EJD566Z1https://board.circlewithadot.net/post/https://come-from.mad-scientist.club/users/algernon/statuses/01KS3GDYJWX7S92CP9EJD566Z1Wed, 20 May 2026 20:15:00 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:14:48 GMT]]>@jedbrown @inthehands I can only go by German/EU law, hand here it is not transformative (becaise duh!). The reproduction is the key thing here: if you reproduce another's work outside of private use, you are violating Urheberrecht (creator's rights): priviledges enshrined in law to the creator of a work (some of which can be licensed out). One of these is distributing reproductions.
E.g. any time you upload an image to SM, their ToS say you grant them license to reproduce (amonh others).

]]>https://board.circlewithadot.net/post/https://social.anoxinon.de/users/haihappen/statuses/116608734894421556https://board.circlewithadot.net/post/https://social.anoxinon.de/users/haihappen/statuses/116608734894421556Wed, 20 May 2026 20:14:48 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:12:58 GMT]]>@algernon @inthehands @joe @ShadSterling Are there any pre-defined templates available except the standard one?

]]>https://board.circlewithadot.net/post/https://mstdn.social/ap/users/116238961427035612/statuses/116608727709064392https://board.circlewithadot.net/post/https://mstdn.social/ap/users/116238961427035612/statuses/116608727709064392Wed, 20 May 2026 20:12:58 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:10:14 GMT]]>@raulmatias @inthehands @joe @ShadSterling I have nothing to implement, iocaine can already do that. The template can be changed, and the random seed doesn't need to be stable, so it can serve different junk for every request.

But! I don't think that (= serving different garbage for every request) is a good idea. That makes it almost trivial to detect that iocaine is at work: send two requests, compare. Wildly different? That's gonna be garbage => engage countermeasures.

Serving the same garbage makes automatic detection a tiny bit harder.

But, in case of Google, that still doesn't matter, as long as they identify themselves.

Personally, I send an x-robots-tag http header that opts it out of all kinds of search and indexing, from crawlers that respect it (Google currently does). I still serve googlebot garbage, but that bot visits me like a handful of times a day.

]]>https://board.circlewithadot.net/post/https://come-from.mad-scientist.club/users/algernon/statuses/01KS3G57FBB9FQQNMM078V0KVNhttps://board.circlewithadot.net/post/https://come-from.mad-scientist.club/users/algernon/statuses/01KS3G57FBB9FQQNMM078V0KVNWed, 20 May 2026 20:10:14 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 20:02:30 GMT]]>@inthehands @joe @ShadSterling Detecting them would be way harder if they served different HTML and layout structures each time, maybe with some of them copied from popular CMSes/forum software/wiki software/any software for pre-generated pages.

@algernon perhaps you'd want to implement this.

]]>https://board.circlewithadot.net/post/https://mstdn.social/ap/users/116238961427035612/statuses/116608686499465950https://board.circlewithadot.net/post/https://mstdn.social/ap/users/116238961427035612/statuses/116608686499465950Wed, 20 May 2026 20:02:30 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:58:57 GMT]]>@raulmatias @inthehands Ooh, good point - I had completely forgotten about screen readers in that context.

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/mathaetaes/statuses/116608672575151830https://board.circlewithadot.net/post/https://infosec.exchange/users/mathaetaes/statuses/116608672575151830Wed, 20 May 2026 19:58:57 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:58:01 GMT]]>@inthehands @philbaker1 I don't think cynicism is surrender. Cynicism is being aware of what could happen and preparing yourself mentally for it. Just because I'm cynical about some things doesn't mean I've given into them. It means I'm aware of them and I know how bad they can get.

]]>https://board.circlewithadot.net/post/https://mastodon.world/users/mlanger/statuses/116608668915094744https://board.circlewithadot.net/post/https://mastodon.world/users/mlanger/statuses/116608668915094744Wed, 20 May 2026 19:58:01 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:53:17 GMT]]>@macronaut @khm @inthehands maybe an appropriate message would be "send 1XMR to be unblocked"...

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/hyc/statuses/116608650288251753https://board.circlewithadot.net/post/https://mastodon.social/users/hyc/statuses/116608650288251753Wed, 20 May 2026 19:53:17 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:51:27 GMT]]>@mathaetaes @inthehands > posting tables as poisoned images rather than text

Please **never** do that. Accessibility is more important than poisoning LLMs.

]]>https://board.circlewithadot.net/post/https://mstdn.social/ap/users/116238961427035612/statuses/116608643096634788https://board.circlewithadot.net/post/https://mstdn.social/ap/users/116238961427035612/statuses/116608643096634788Wed, 20 May 2026 19:51:27 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:48:51 GMT]]>@inthehands I fear that when we block the bot, they'll access the site through Chrome. Getting what people browsing get. I don't know what could stop them.

]]>https://board.circlewithadot.net/post/https://ludosphere.fr/users/gherhartd/statuses/116608632823193552https://board.circlewithadot.net/post/https://ludosphere.fr/users/gherhartd/statuses/116608632823193552Wed, 20 May 2026 19:48:51 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:42:41 GMT]]>@ShadowJonathan @inthehands they alsof are pretty involved with the contents of the standard (75% of the authors), so luring the crawler into a pit of crappy data is probably your only way to protest besides avoiding to hand any of your money and attention to them.

Link: https://datatracker.ietf.org/doc/html/rfc9309

]]>https://board.circlewithadot.net/post/https://mastodon.nl/users/wsslmn/statuses/116608608592812094https://board.circlewithadot.net/post/https://mastodon.nl/users/wsslmn/statuses/116608608592812094Wed, 20 May 2026 19:42:41 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:35:38 GMT]]>@inthehands thank you friend. Adding your quote to my common place book

]]>https://board.circlewithadot.net/post/https://speedlines.stctp.zone/users/Leneux/statuses/116608580897137260https://board.circlewithadot.net/post/https://speedlines.stctp.zone/users/Leneux/statuses/116608580897137260Wed, 20 May 2026 19:35:38 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:29:40 GMT]]>@khm

❤

@inthehands

]]>https://board.circlewithadot.net/post/https://mastodon.xyz/users/401matthall/statuses/116608557445406164https://board.circlewithadot.net/post/https://mastodon.xyz/users/401matthall/statuses/116608557445406164Wed, 20 May 2026 19:29:40 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:28:43 GMT]]>@inthehands If I understand your question correctly (sorry if it's not the case) I think that Anubis, the AI crawler protection, could be part of the solution. Not only would that work for Google, that would (or at least *should*) also work against other crawlers.
Another advantage is that it can work along your other solutions.
OTHO the drawback is that it would work against all crawler, so you would "disappear" from every search engine... 🤔

Link Preview Image
Anubis (software) - Wikipedia

favicon

(en.wikipedia.org)

]]>https://board.circlewithadot.net/post/https://climatejustice.social/users/gturri/statuses/116608553706520719https://board.circlewithadot.net/post/https://climatejustice.social/users/gturri/statuses/116608553706520719Wed, 20 May 2026 19:28:43 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:25:17 GMT]]>@albertcardona @inthehands It involves a couple steps, given the idiosyncrasies of the nginx regex support (no full pcre here!).
I keep two classes of blocked agents: (1) bad agents; and (2) scrapping false agents. A third regex unblocks agents that are false positives (due to (2)).

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/schamschula/statuses/116608540202191733https://board.circlewithadot.net/post/https://mastodon.social/users/schamschula/statuses/116608540202191733Wed, 20 May 2026 19:25:17 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:24:26 GMT]]>@RnDanger @inthehands well, we don’t know and we will see. My guess are separate scrapers (officially) and a lot of mistrust (are there others?) and masses of unidentified scrapers. Nevertheless, Google can better afford to play by the rules, since hey already own the largest index. Think also of Video etc. Will volume win the war? Or quality and freshness? Etc. Future is difficult.

]]>https://board.circlewithadot.net/post/https://nrw.social/users/korrupt/statuses/116608536826849063https://board.circlewithadot.net/post/https://nrw.social/users/korrupt/statuses/116608536826849063Wed, 20 May 2026 19:24:26 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:22:19 GMT]]>@inthehands Two quotes from Pratchett comes to mind

>>> “All witches are selfish, the Queen had said. But Tiffany’s Third Thoughts said: Then turn selfishness into a weapon! Make all things yours! Make other lives and dreams and hopes yours! Protect them! Save them! Bring them into the sheepfold! Walk the gale for them! Keep away the wolf! My dreams! My brother! My family! My land! My world! How dare you try to take these things, because they are mine!

>>> "We look to ... the edges," said Mistress Weatherwax. "There's a lot of edges, more than people know. Between life and death, this world and the next, night and day, right and wrong ... an' they need watchin'. We watch 'em, we guard the sum of things. And we never ask for any reward. That's important.”

]]>https://board.circlewithadot.net/post/https://hachyderm.io/users/Di4na/statuses/116608528535023213https://board.circlewithadot.net/post/https://hachyderm.io/users/Di4na/statuses/116608528535023213Wed, 20 May 2026 19:22:19 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:20:59 GMT]]>@inthehands I am all in on reciprocating... what do we do?

]]>https://board.circlewithadot.net/post/https://types.pl/users/agentultra/statuses/116608523295950060https://board.circlewithadot.net/post/https://types.pl/users/agentultra/statuses/116608523295950060Wed, 20 May 2026 19:20:59 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:20:49 GMT]]>@khm @inthehands I'm more interested in sending it to AI scrapers, not that they tend to identify themselves honestly. 🙄

]]>https://board.circlewithadot.net/post/https://mspsocial.net/users/jima/statuses/116608522584726689https://board.circlewithadot.net/post/https://mspsocial.net/users/jima/statuses/116608522584726689Wed, 20 May 2026 19:20:49 GMT<![CDATA[Reply to Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because on Wed, 20 May 2026 19:20:16 GMT]]>@markwyner @inthehands
There is a point where their search becomes bad enough that being on Google search has less and less payoff

]]>https://board.circlewithadot.net/post/https://mastodon.social/users/AccordionBruce/statuses/116608520473710840https://board.circlewithadot.net/post/https://mastodon.social/users/AccordionBruce/statuses/116608520473710840Wed, 20 May 2026 19:20:16 GMT