To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
-
@osm_tech how can you even scrape a mapsite? isnt osm just a big canvas like google maps that is also nearly impossible to automate?
@utf_7 It is madness, start here: https://www.openstreetmap.org/node/1 and keep going once you reach https://www.openstreetmap.org/node/10000000000, then start on ways, and relations
or just download the latest weekly export from planet.openstreetmap.org 
-
@utf_7 It is madness, start here: https://www.openstreetmap.org/node/1 and keep going once you reach https://www.openstreetmap.org/node/10000000000, then start on ways, and relations
or just download the latest weekly export from planet.openstreetmap.org @osm_tech uff, i am a noob so forgive my stupid question: cant you somehow limit the requests. like 10 requests per minute or so. normal users will not be affected and scrapers will take forever?
-
@HunterZ @osm_tech this is actually quite common. Mobile advertising SDKs for games, background apps, etc include residential scraping proxy functionality that they can sell to the highest bidder, and then when scrapers want to avoid restrictions they can pay a fraction of a penny to send their requests via your phone. Millions of people use apps with this built in and have no idea. Most websites don't want to ban the residential scrapers because it can hurt growth.
@ryanprior @HunterZ @osm_tech I had no idea this was a thing. And presumably, as requests come from you, not the advertiser, Pihole (and other network blockers) treat it as legitimate traffic?
-
@osm_tech uff, i am a noob so forgive my stupid question: cant you somehow limit the requests. like 10 requests per minute or so. normal users will not be affected and scrapers will take forever?
@utf_7 We've had 400,000 IPs in the last 24 hours. Each IP only does a few requests. Technically we're managing, but no fun fighting this daily rather than building new things.
-
@ryanprior @HunterZ @osm_tech I had no idea this was a thing. And presumably, as requests come from you, not the advertiser, Pihole (and other network blockers) treat it as legitimate traffic?
-
@ryanprior @HunterZ @osm_tech I had no idea this was a thing. And presumably, as requests come from you, not the advertiser, Pihole (and other network blockers) treat it as legitimate traffic?
@tehstu @ryanprior @osm_tech pihole works by refusing to provide DNS resolution for domains on its blocklists, so it could block a scraper *if* its functionality depends on resolving a domain name that is blocked by pihole.
-
@tehstu @ryanprior @osm_tech pihole works by refusing to provide DNS resolution for domains on its blocklists, so it could block a scraper *if* its functionality depends on resolving a domain name that is blocked by pihole.
@tehstu @ryanprior @osm_tech oh and of course the scraper would have to respect pihole versus using its own hard coded DNS IP to resolve things.
-
@utf_7 We've had 400,000 IPs in the last 24 hours. Each IP only does a few requests. Technically we're managing, but no fun fighting this daily rather than building new things.
@osm_tech tHeN yOu jUsT neEd tO sCaLe
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse@osm_tech question. Why do people scrape server which make the data freely available? And, probably, better structured in the final product. I don't see the point.
-
@osm_tech question. Why do people scrape server which make the data freely available? And, probably, better structured in the final product. I don't see the point.
@JonSaenzAgirre It is a good questions, and we don't know the answer either. Our planet data is so much easier to process and use.
-
@osm_tech tHeN yOu jUsT neEd tO sCaLe
@utf_7 In this economy with RAM prices what they are?!?
-
@osm_tech question. Why do people scrape server which make the data freely available? And, probably, better structured in the final product. I don't see the point.
@JonSaenzAgirre vibe coders gonna vibe
-
@osm_tech does coming from residential IPs mean that someone has baked a scraper into some popular tool that people don't realize is doing that?
@HunterZ@mastodon.sdf.org @osm_tech@en.osm.town lots of mobile/desktop apps, browser extensions, and even IoT devices are paid by "residential proxy" companies to prey on their users by selling said users's connections to AI scrapers https://www.spamhaus.org/resource-hub/compromised/lets-talk-about-the-danger-of-residential-proxy-networks/
-
@HunterZ @osm_tech this is actually quite common. Mobile advertising SDKs for games, background apps, etc include residential scraping proxy functionality that they can sell to the highest bidder, and then when scrapers want to avoid restrictions they can pay a fraction of a penny to send their requests via your phone. Millions of people use apps with this built in and have no idea. Most websites don't want to ban the residential scrapers because it can hurt growth.
@ryanprior @HunterZ @osm_tech I have that scraping also on my private webserver and it forced me to make a whole bunch of content private. yet still the botnet scrapes onto it and gets 404s now. Every single request from a different IP...
-
@ryanprior @HunterZ @osm_tech I have that scraping also on my private webserver and it forced me to make a whole bunch of content private. yet still the botnet scrapes onto it and gets 404s now. Every single request from a different IP...
@olbohlen @HunterZ @osm_tech sad to hear that! It's wild though, you can sign up for a scraper proxy service in minutes. They're legal, inexpensive, and easy to use. Admins who assume scrapers are using their own machines that inauthentic traffic will come from a few IP addresses are sadly living in the past.
-
@olbohlen @HunterZ @osm_tech sad to hear that! It's wild though, you can sign up for a scraper proxy service in minutes. They're legal, inexpensive, and easy to use. Admins who assume scrapers are using their own machines that inauthentic traffic will come from a few IP addresses are sadly living in the past.
@ryanprior @HunterZ @osm_tech sure I could, but I refuse to put my selfhosted stuff behind some new dependency...
-
@ryanprior @HunterZ @osm_tech sure I could, but I refuse to put my selfhosted stuff behind some new dependency...
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #AbuseMight be a good idea to become OSMF Member now or just donate some money.
Membership is starting at 15Β£/yer
https://supporting.openstreetmap.org/ -
@JonSaenzAgirre It is a good questions, and we don't know the answer either. Our planet data is so much easier to process and use.
@osm_tech @JonSaenzAgirre thats dumb ai, probably. No "i" at all...
-
To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.
If you need OSM data, please don't scrape the website - use the official downloads at https://planet.openstreetmap.org
#AI #Bots #Abuse@osm_tech sounds familiar, last year I braved turning cloudflares "under attack" mode off for https://dnshistory.org/ and saw an extra 5 million requests/day (500k unique IPs) overloading things. It's still blocking >700k requests/day a month later...
