If you self-host services on the internet, you may have seen waves of crawlers hammering your websites without mercy.

[algernon@come-from.mad-scientist.club]

@Soblow Thank you for writing this, reading it has been very educational!

I have a number of things in development that will make iocaine less of a pain, and more suitable for cases like yours - but... that takes a bit of time.

And NSoE's documentation is a mess, indeed. My excuse is that it was never meant to be used by anyone else, it's something I wrote for me. But there was no other option for a long time, and even if iocaine3 has a built-in script now, that's not as good as NSoE (yet).

I have plans to address that shortcoming, so there's an option that isn't NSoE, has useful, navigatable documentation that isn't written like a mad scientist's diary¹.

But all the issues you listed are valid, and you even highlighted shortcomings I wasn't aware of, and tricks I did not consider. Now I have more things to play with!

---

1. That was highly amusing to read, and I chuckled. Thanks!² ︎

2. Yes, I know, it's not a praise, but... algernon looks at the domain he's tooting from... yeah. ︎

[dj2mn@mastodon.sdf.org]

@Soblow OMG that's horrendous. I'm too scared to look at my (recently set up) nginx logs now.

[dfxluna@transfem.social]

@Soblow@eldritch.cafe your prose is so clear and easy to follow! I love it :3

[ottermatic@woof.group]

@Soblow thank you loved reading this.

[khleedril@cyberplace.social]

@Soblow Owch. Painful long read, regurgitating the experience of a lot of tech administrators. I like the idea of poisoning AI, but giving them an infinite pile of garbage is a waste of my resources. I'll leave that to people like you

ps. Please learn the difference between scrapping and scraping, and dairy and diary.

[soblow@eldritch.cafe]

@booklordofthedings Oh, for bookmark sync, I wouldn't recommend Linkding, but maybe something like Floccus (and maybe Nextcloud bookmarks)

[soblow@eldritch.cafe]

@khleedril Oops, thanks for your feedback!
(english isn't my native language )

[soblow@eldritch.cafe]

@algernon Thanks for your reply!

It may not be obvious, but I don't consider "mad scientist" as a depreciative term so yeah (and to be honest, I didn't even realize it was in your domain name )

The main struggles I had were because both iocaine and NSoE use languages I'm not used to, which isn't your fault.

Again, your software work nicely and I'm glad they exist and work!
(And I know the problem of "this was meant to be a solution for me that I made available to everyone", again it's fully understandable and, that's open-source software, it comes with no expectation nor warranty ).

[flann@boosts.squirrel.pictures]

@Soblow Good post!

A lot more comprehensive than my own, I went more with just quickly highlighting a few special cases rather than laying out the entire setup

If you don't mind I'll link to your post alongside the one from lux that was already there.

[soblow@eldritch.cafe]

@flann Please do
I'll go read that post when I have time~

[soblow@eldritch.cafe]

As promised, I updated the linked repository to add a README (and a license)

[soblow@eldritch.cafe]

@flann Okay, I read it, it's highly interesting and it would've helped if I saw this earlier

There are things from your blogpost I'll likely try later too

[pandro@fedi.imowl.net]

@Soblow@eldritch.cafe
A nice read! Nothing too new for me as I was following you live on that journey but good to hear you found something that helps!

Also using iocaine for my services I at one point thought "why not let iocaine also serve garbage to empty user-agents?". That'd also catch a lot of the vuln scanners that are convinced I'm using wordpress (I'm not).

There's a surprising number of legitimate traffic you wouldn't expect to not set a UA ​​

[soblow@eldritch.cafe]

@pandro That could be something, but for example the "fursona lookup" tool didn't have a User Agent set (until I told the author)...

[seldenotter@blimps.xyz]

@Soblow "I spent the last few years building up a tolerance to iocaine powder."

[icewolf@masto.brightfur.net]

@Soblow Hah... I think we're getting iocaine or something when trying to read the article on our phone (iOS Safari, on iOS 15 or something like that). Haven't tried our desktop. Pretty meta, though.

[bandie@yip.gay]

@Soblow Hmmm.. I think of serving a knowledge.tld just with static garbage now...

[soblow@eldritch.cafe]

Well, first documented case (on my end) of a false positive, yay...

iOS 15 + Safari, for some reason...

[jernej__s@infosec.exchange]

@Soblow Interesting, I've noticed a very similar pattern on a client's web server, which they use for hosting internal projects (which have to be available publicly) – 5000 requests per second every few hours on specific subdomains, from residential IPs, with each IP doing about 20 requests, changing the user-agent once. Nothing knowledge-like though, most of the sites run Wordpress.

[sargeros@rivals.space]

@Soblow just tried to add your article to my self hosted Readeck (a read it later service) and I think the request got caught by an anti scraper.
Guess I'll have to read the article soon to understand what happened!