Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
@raulmatias @inthehands @joe @ShadSterling I have nothing to implement, iocaine can already do that. The template can be changed, and the random seed doesn't need to be stable, so it can serve different junk for every request.
But! I don't think that (= serving different garbage for every request) is a good idea. That makes it almost trivial to detect that iocaine is at work: send two requests, compare. Wildly different? That's gonna be garbage => engage countermeasures.
Serving the same garbage makes automatic detection a tiny bit harder.
But, in case of Google, that still doesn't matter, as long as they identify themselves.
Personally, I send an
x-robots-taghttp header that opts it out of all kinds of search and indexing, from crawlers that respect it (Google currently does). I still serve googlebot garbage, but that bot visits me like a handful of times a day.@algernon @inthehands @joe @ShadSterling Are there any pre-defined templates available except the standard one?
-
@inthehands It's important to note that search indexing is considered "transformative" and thus fair use *because* it does not supplant the market for the original content. That goes out the window when the product functions to capture traffic that would otherwise go to the cites. They are acting with impunity, but existing copyright law addresses this if courts find it to be not transformative.
@jedbrown @inthehands I can only go by German/EU law, hand here it is not transformative (becaise duh!). The reproduction is the key thing here: if you reproduce another's work outside of private use, you are violating Urheberrecht (creator's rights): priviledges enshrined in law to the creator of a work (some of which can be licensed out). One of these is distributing reproductions.
E.g. any time you upload an image to SM, their ToS say you grant them license to reproduce (amonh others). -
@algernon @inthehands @joe @ShadSterling Are there any pre-defined templates available except the standard one?
@raulmatias @inthehands @joe @ShadSterling There's the built-in one, and another - slightly more complex - in Nam-Shub of Enki.
There will be more templates coming in the next few months (and new scripts!).
But a lot of things are doable today, if someone takes the time and creates a suitable template.
-
Going with meta noindex for now. My thinking is that this actively tells Google to yank already-crawled content from their index, whereas they might take a robots.txt entry to mean “do not update, but keep showing last fetched.”
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
-
instead of no-index ―because this would affect all search engines, not just Google― isn’t there a way to target Google specifically in robots.txt?
there should be a list of all the major techbros crawlers ―Google, Microslop, Facebook, Amazon, X, etc.
@blogdiva
I believe that my various name=“___” values specifically target Google.Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
-
@blogdiva
I believe that my various name=“___” values specifically target Google.Based on what I’ve read, blocking them in robots.txt will only stop them from •updating• their scrape, whereas noindex means “do not use.” (I have long blocked their LLM-specific bots in robots.txt.)
@inthehands TIL thanks
-
@inthehands TIL thanks
Keep it in pencil. I’m still learning myself, and not sure I understand everything correctly here.
