Pleased to share a page and explainer for the AI tarpit project Science is Poetry, with legal statement, rationale(s), and a few deployment notes:
-
@JulianOliver I think scraper bots and other parasites constantly scan TLS transparency reports to find new domains to probe. As soon as you have a new certificate, they start hitting your web server.
@pertho Very interesting! I will look into this closely. Thank you.
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver heya. How do I donate and configure domains for your experiment? Do they come with any restrictions, i.e. must the FQDNs be in distinct DNS zones, etc? Can easily spin up and configure a few to begin with e.g.
poetry.zenr.io
worst.case.zenr.io
wurst.case.zenr.io
et.c. -
@JulianOliver @futuresprog ah, cool to know, thanks. can config a few from this ...
-
@JulianOliver @futuresprog ah, cool to know, thanks. can config a few from this ...
@vortex @futuresprog Thanks a lot Adam! Please let me know when you're done. By DM is also fine too.
-
@JulianOliver I think scraper bots and other parasites constantly scan TLS transparency reports to find new domains to probe. As soon as you have a new certificate, they start hitting your web server.
@pertho @JulianOliver a way around this is to use both DNS and TLS wildcard support - there is no single domain list in the report that can be slopped.
-
@pertho @JulianOliver a way around this is to use both DNS and TLS wildcard support - there is no single domain list in the report that can be slopped.
@dch I would think they would still try the bare/apex domain and "www".
@JulianOliver -
@pertho Very interesting! I will look into this closely. Thank you.
@JulianOliver @pertho yes this is the thing. you can get hold of live streams of newly created TLS certificates, an example https://bencevans.io/security/certificate-stream
-
I've started to harvest a list of AI crawler endpoint addrs for your blacklisting pleasure.
I'll try to keep it updated. I've been fastidious with ensuring I'm only pulling those related to the known user agent, so as not to have any false positives
https://scienceispoetry.net/files/parasites.txt
It is at the same path for all contributed domains.
For instance:
@JulianOliver "parasites" is a great name for this
-
Pleased to share a page and explainer for the AI tarpit project Science is Poetry, with legal statement, rationale(s), and a few deployment notes:
The page may grow a bit. Just wanted to get it out the door.
@JulianOliver do you want some subdomains on https://nein.wtf? feel free to pick!
-
@JulianOliver do you want some subdomains on https://nein.wtf? feel free to pick!
-
I've started to harvest a list of AI crawler endpoint addrs for your blacklisting pleasure.
I'll try to keep it updated. I've been fastidious with ensuring I'm only pulling those related to the known user agent, so as not to have any false positives
https://scienceispoetry.net/files/parasites.txt
It is at the same path for all contributed domains.
For instance:
It's approaching DoS at this point. This just one of the VMs, and just OpenAI's parasite.
Threading's holding up but need some more tuning of rate limits and burst. Trying sending 429's now to ask them to play nice.
To think the www was built for people.
And here we are
-
@JulianOliver they are, A and AAAA records already set up
-
@JulianOliver they are, A and AAAA records already set up
@computersandblues Ahh, so good. I'll add them first thing tomorrow. It's evening here. Will let you know as soon as I do
-
@dch I would think they would still try the bare/apex domain and "www".
@JulianOliver@pertho in my experience it seems they don’t even bother with wildcard domains @JulianOliver
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver I have a slew of currently un -used domains. Where do I point them?
-
@JulianOliver I have a slew of currently un -used domains. Where do I point them?
@WTL Thanks! Just these two records:
A: 95.216.76.85
AAAA: 2a01:4f9:2b:c83::2 -
I've started to harvest a list of AI crawler endpoint addrs for your blacklisting pleasure.
I'll try to keep it updated. I've been fastidious with ensuring I'm only pulling those related to the known user agent, so as not to have any false positives
https://scienceispoetry.net/files/parasites.txt
It is at the same path for all contributed domains.
For instance:
Hi @JulianOliver,
mind adding a parallel https://cdb.cr.yp.to file? -
It's approaching DoS at this point. This just one of the VMs, and just OpenAI's parasite.
Threading's holding up but need some more tuning of rate limits and burst. Trying sending 429's now to ask them to play nice.
To think the www was built for people.
And here we are
Hi @JulianOliver,
indeed an act of #hygiene blocking #bots: https://doi.org/10.17487/RFC8890 "The Internet is for End Users" -
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
You can add `dreckiger.schleimpilz.ch` to the list. Thanks for all your work!
-
Do you have an unused domain that you would be happy to donate to a counter-offensive against unchecked & unregulated AI crawlers that scrape human-made content to simulate & deceive for profit?
If so, pls reply to this post. Your domain would become an entrypoint to the AI tarpit & Poison-as-a-Service project below, allowing concerned public to choose to use it on their sites, helping make the project more resilient to blacklisting.
@JulianOliver I have a handful of domains that I would love to see used for tarpitting.
