curl.se average bandwidth served per second (by #Fastly) over the last month

[bagder@mastodon.social]

curl.se average bandwidth served per second (by #Fastly) over the last month

244 Mbps on average. Peak at 293 Mbps.

A total of about 76 terabytes.

[henrik@eliitin-some.fi]

@bagder

Wow... What is the most requested item? Some sort of Torrent like distribution should be in order.

[bagder@mastodon.social]

@henrik you then assume curl downloads is a significant portion of this, and it really isn't. Roughly %0.01 of the requests are tarball downloads.

[jackryder@infosec.exchange]

@bagder agents just going insane?

It seems like every tool is now trying to hit every website all the time in an effort to somehow become the first to find the new data.

[bagder@mastodon.social]

@jackryder I don't know. I don't have logs, only some accumulated numbers.

[rallias@hax.social]

@bagder I realize this is probably an unattainable figure, but there's a small part of me that's curious how much of that traffic comes from user agents using curl

[bagder@mastodon.social]

@rallias as I don't have logs I can't tell, but some of the accumulated stats clearly indicate that a large share of the traffic is bots - not humans with web browsers.

[jackryder@infosec.exchange]

@bagder With having no more information than everyone else... it certainly appears that you are extra-popular.

Have you seen tools like Iocaine? I kind of assume they are known but I don't like to assume.

https://iocaine.madhouse-project.org/

The argument, I believe, is to make the scraping too expensive/worthless for the bots.

[simonzerafa@infosec.exchange]

@bagder

That seems like a lot?

Do you get issues with automated downloads breaking and contributing to repeated downloads unnecessarily?

Do you have mitigations in place to deal with any abuse similar to that?