83% of requests (globally) to our apex homepages (bbc.co.uk/ and bbc.com/) are from "Python Requests".

[viss@mastodon.social]

@tdp_org have you checked out the source ips to see if its just a bunch of cloud vms or something?

[tdp_org@mastodon.social]

@Viss There's quarter of a million distinct IPs so no, I haven't done that but I have looked at the ASNs from which the requests come and they're mostly eyeball ISPs. I suspect there's some common router config or similar which uses it as "is the internet working" - the usual

[viss@mastodon.social]

@tdp_org ah, could possibly be botnet type stuff with all the edge devices that have been popped recently and turned into proxies?

[tdp_org@mastodon.social]

@Viss It's possible but it's so consistent that I doubt it.
One of the many "joys" of being a relatively well-known presence on the internet I spose

[viss@mastodon.social]

@tdp_org what are all those python clients actually requesting? is it just the same uri over and over again from 250k hosts?

[simonzerafa@infosec.exchange]

@tdp_org @Viss

Maybe change the the domain to:

BritishBroadcastingCorporation.co.uk

So they have to take a long longer to type it into a script?

Any plans to block or otherwise deal with that traffic?

[relet@flipping.rocks]

@tdp_org

What will be the impact on critical infrastructure if you decide to drop or reject all the connectivity checks from one day to another?

Care to find out?

[tdp_org@mastodon.social]

@Gaelan Yeah we're on plenty of those lists
I suppose it's a compliment of sorts

[jamesog@mastodon.social]

@tdp_org Default user agents are the bane of my life. I wish libraries would force users to set it correctly

[doctormo@floss.social]

@tdp_org @Viss

Possible. I run inkscape's website and it's shocking what my ASN tools show. Some of the IPs slam requests, others hit with less frequency. But ASN plus the honey trap gets a decent set of addresses to block or degrade. (Our poor machine can't handle CDN levels of requests)