Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
Mind sharing the necessary subset of the nginx config to enforce robots.txt as an nginx block list? Thank you.
@albertcardona @inthehands It involves a couple steps, given the idiosyncrasies of the nginx regex support (no full pcre here!).
I keep two classes of blocked agents: (1) bad agents; and (2) scrapping false agents. A third regex unblocks agents that are false positives (due to (2)). -
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands If I understand your question correctly (sorry if it's not the case) I think that Anubis, the AI crawler protection, could be part of the solution. Not only would that work for Google, that would (or at least *should*) also work against other crawlers.
Another advantage is that it can work along your other solutions.
OTHO the drawback is that it would work against all crawler, so you would "disappear" from every search engine...
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands thank you friend. Adding your quote to my common place book
-
@inthehands this is a fence-post defense against this, google Will Not Care
just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output
@ShadowJonathan @inthehands they alsof are pretty involved with the contents of the standard (75% of the authors), so luring the crawler into a pit of crappy data is probably your only way to protest besides avoiding to hand any of your money and attention to them.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands I fear that when we block the bot, they'll access the site through Chrome. Getting what people browsing get. I don't know what could stop them.
-
@inthehands I know of at least one professional artist who has deliberately poisoned their images, in an attempt to deter AI scraping (mostly because the scrapers blast her small site and effectively DoS it). If they follow robots.txt, they're not affected... but they were already ignoring robots.txt
I just read an IARPA paper that said poisoning as little as .1% of training data can disrupt a model. If content creators choose to deliberately poison content that they ask not to be scraped, it might be a nice way to deter bad behavior.
The tools I know of work on imagery, but with effort people may come up with stuff that works on data as well. E.g., burying base64-encoded malicious prompts in your text, posting tables as poisoned images rather than text, etc.
Seems like we should start organizing and taking firm action now, before AI companies start buying politicians and making such defenses illegal.
@mathaetaes @inthehands > posting tables as poisoned images rather than text
Please **never** do that. Accessibility is more important than poisoning LLMs.
-
@macronaut @khm @inthehands the update would change that 403 to a 402. And add "error_page 402 /402.html;" to the server{} config, and create the /402.html file in the docroot containing whatever desired message.
@macronaut @khm @inthehands maybe an appropriate message would be "send 1XMR to be unblocked"...
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands @philbaker1 I don't think cynicism is surrender. Cynicism is being aware of what could happen and preparing yourself mentally for it. Just because I'm cynical about some things doesn't mean I've given into them. It means I'm aware of them and I know how bad they can get.
-
@mathaetaes @inthehands > posting tables as poisoned images rather than text
Please **never** do that. Accessibility is more important than poisoning LLMs.
@raulmatias @inthehands Ooh, good point - I had completely forgotten about screen readers in that context.
-
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
@inthehands @joe @ShadSterling Detecting them would be way harder if they served different HTML and layout structures each time, maybe with some of them copied from popular CMSes/forum software/wiki software/any software for pre-generated pages.
@algernon perhaps you'd want to implement this.
-
@inthehands @joe @ShadSterling Detecting them would be way harder if they served different HTML and layout structures each time, maybe with some of them copied from popular CMSes/forum software/wiki software/any software for pre-generated pages.
@algernon perhaps you'd want to implement this.
@raulmatias @inthehands @joe @ShadSterling I have nothing to implement, iocaine can already do that. The template can be changed, and the random seed doesn't need to be stable, so it can serve different junk for every request.
But! I don't think that (= serving different garbage for every request) is a good idea. That makes it almost trivial to detect that iocaine is at work: send two requests, compare. Wildly different? That's gonna be garbage => engage countermeasures.
Serving the same garbage makes automatic detection a tiny bit harder.
But, in case of Google, that still doesn't matter, as long as they identify themselves.
Personally, I send an
x-robots-taghttp header that opts it out of all kinds of search and indexing, from crawlers that respect it (Google currently does). I still serve googlebot garbage, but that bot visits me like a handful of times a day. -
@raulmatias @inthehands @joe @ShadSterling I have nothing to implement, iocaine can already do that. The template can be changed, and the random seed doesn't need to be stable, so it can serve different junk for every request.
But! I don't think that (= serving different garbage for every request) is a good idea. That makes it almost trivial to detect that iocaine is at work: send two requests, compare. Wildly different? That's gonna be garbage => engage countermeasures.
Serving the same garbage makes automatic detection a tiny bit harder.
But, in case of Google, that still doesn't matter, as long as they identify themselves.
Personally, I send an
x-robots-taghttp header that opts it out of all kinds of search and indexing, from crawlers that respect it (Google currently does). I still serve googlebot garbage, but that bot visits me like a handful of times a day.@algernon @inthehands @joe @ShadSterling Are there any pre-defined templates available except the standard one?
-
@inthehands It's important to note that search indexing is considered "transformative" and thus fair use *because* it does not supplant the market for the original content. That goes out the window when the product functions to capture traffic that would otherwise go to the cites. They are acting with impunity, but existing copyright law addresses this if courts find it to be not transformative.
@jedbrown @inthehands I can only go by German/EU law, hand here it is not transformative (becaise duh!). The reproduction is the key thing here: if you reproduce another's work outside of private use, you are violating Urheberrecht (creator's rights): priviledges enshrined in law to the creator of a work (some of which can be licensed out). One of these is distributing reproductions.
E.g. any time you upload an image to SM, their ToS say you grant them license to reproduce (amonh others). -
@algernon @inthehands @joe @ShadSterling Are there any pre-defined templates available except the standard one?
@raulmatias @inthehands @joe @ShadSterling There's the built-in one, and another - slightly more complex - in Nam-Shub of Enki.
There will be more templates coming in the next few months (and new scripts!).
But a lot of things are doable today, if someone takes the time and creates a suitable template.
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
-
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
@blogdiva
I believe that my various name=“___” values specifically target Google.Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
-
@blogdiva
I believe that my various name=“___” values specifically target Google.Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
@inthehands TIL thanks
-
@inthehands TIL thanks
Keep it in pencil. I’m still learning myself, and not sure I understand everything correctly here.
