Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
-
OK, a •lot• of replies need this reponse:
Yes, of •course• they will start ignoring robots.txt etc as soon as they think it hurts their business. Of course.
It is important to •force that fight•, rather than just capitulating in advance.
@inthehands If they ignore robots.txt, they will be added to the block list in nginx.conf. My robots.txt has a note stating as much. There is plenty of company there!
-
@dotstdy You saw what happened to the last guy who did that, right?
*blue flash*
@sennoma @joe @inthehands @ShadSterling it's okay i put an `unsafe` label on the door
-
R relay@relay.publicsquare.global shared this topic
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands I think if you really want to sell that point, you should explicitly disallow googlebot in robots.txt, then also setup the middleware to respond with 404 to any other URI requested by googlebot
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
I return402 Payment Requiredto googlebot user agents
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
@inthehands hard agree (even though I'm thinking of this in broader terms). It can be so difficult to keep the will to do so, in practice... but it's important to do it whenever we can.
-
I return
402 Payment Requiredto googlebot user agents@khm @inthehands Oooh. I like it. Gonna have to add that to my nginx configs instead of 403. And add a Monero wallet address in the text content
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands this is a great point
-
@khm @inthehands Oooh. I like it. Gonna have to add that to my nginx configs instead of 403. And add a Monero wallet address in the text content
@hyc @khm @inthehands is there a “how to” on this that one can use to update their web server/site?
-
@inthehands this is a fence-post defense against this, google Will Not Care
just start poisoning the data once you detect that google is the one fetching it, just absolutely fucking destroy their LLM output
@ShadowJonathan @inthehands agree. I don't think defense is the best reaction to sustain a healthy internet. this rhetoric has been untrue since... Google (other similar corps).
random offensive approach such as collective data poisoning, public exposé, factual based journalism, education, jailtime, guillotine & other accountability and positive encouragement should coexist to foster the internet to recover better
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands How about blocking port 80 & 443 access from Google's netblocks? 142.250.0.0/15 for starters.
-
@hyc @khm @inthehands is there a “how to” on this that one can use to update their web server/site?
@macronaut @khm @inthehands Currently I've created in nginx/conf/server_extra/block-useragent.conf:
if ($http_user_agent ~* meta-externalagent) {
return 403;
}And I've added an
include server_extra/*.conf;in my site's server{} config.
-
@macronaut @khm @inthehands Currently I've created in nginx/conf/server_extra/block-useragent.conf:
if ($http_user_agent ~* meta-externalagent) {
return 403;
}And I've added an
include server_extra/*.conf;in my site's server{} config.
@macronaut @khm @inthehands the update would change that 403 to a 402. And add "error_page 402 /402.html;" to the server{} config, and create the /402.html file in the docroot containing whatever desired message.
-
Defeatism is form of surrender. Cynicism is surrender. Despair is surrender. Nihilism is surrender.
Our job is to •care• and to •keep caring• and to •keep doing and keep building• and to •endure• longer than them.
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands Dissallow in robots.txt and install iocaine or anubis or another AI-poisoning software, should they ignore the robots.txt.
-
@cceckman The contract I thought I was signing was this: I published my stuff on a worldwide information network, with no controls whatever, specifically so that anyone anywhere could access it. I did that with full understanding that it would enable people I might not like to read, copy, and share it and put it to uses that I couldn't foresee and might not approve of. And if I didn't want to entertain that possibility I should not have installed a program on my computer whose sole purpose was to deliver of my stuff to any rando who asked for it.
I'm not saying I got a good deal, or that I'm happy with the outcome. But I'm not going to pretend I was tricked or that Google reneged on a bargain. We had no bargain. I served them the stuff anyway, whenever they asked for it.
And I'm not sure I believe Paul Cantrell when he says he thought the contract was different from what I said.
-
@hyc @khm @inthehands is there a “how to” on this that one can use to update their web server/site?
in nginx I have thisif ($http_user_agent ~* (uptime|bot|index|spider|wler|brave)) { return 402 "Just send the money"; }it keeps out the riffraff.
CC: @hyc@mastodon.social @inthehands@hachyderm.io
-
Quick strategy discussion, for those who understand Google indexing and SEO:
If I want to yank a web site out of Google’s now-fully-extractive search, should I (1) disallow googlebot in robots.txt or (2) add `<meta name="googlebot" content="noindex">` to all the page headers?
The goal here is not just to remove my contributions to the commons from Google’s results, but to •make Google aware• that sites are pulling consent. What will best do that?
2/2
@inthehands They're just going to take it anyway.
-
RE: https://tldr.nettime.org/@tante/116605858023186072
Google Search rests on a social contract: their bots can crawl our sites, they can index our sites, and they can show excerpts of our sites because
and •only because•
they send people to our sites. •Our• sites, our words, with our design, with our links, with our context and our aesthetics, shared the way we want to share them.
Google is announcing — unambiguously and with great fanfare — that are fully breaking that contract. We should reciprocate.
1/2
@inthehands Reciprocate how?
-
@macronaut @khm @inthehands Currently I've created in nginx/conf/server_extra/block-useragent.conf:
if ($http_user_agent ~* meta-externalagent) {
return 403;
}And I've added an
include server_extra/*.conf;in my site's server{} config.
@hyc@mastodon.social @macronaut@mas.to @khm@hj.9fs.net @inthehands@hachyderm.io
I redirect those who make some requests either to FSB or to CIA depending on particular paths.
Sadly, I don't think scripts making those requests follow redirects, but those who request .php files which I don't have on my server get redirected to FSB website
-
@joe @ShadSterling
I share Joe’s concern that poison-in-box systems will become detectable, but they seem like a good place to start.I’m even more a fan of bespoke one-off poison generators for those of us who have the means to write them. Both/and.
@inthehands @joe @ShadSterling
Google went out on May 15th and said in their new "spam policy" (seemingly as a preemption) that they will downrank or completely delist sites that try to mess with their AI. Primarily SEO related but I suspect it also extends to "getting it to say stuff" and general poisoning.
I've thought about putting my poison on noindex pages only in order to keep regular search clean and encourage a safe-haven. Might be pointless now.
marginalia-search is pretty nice.
