To keep #OpenStreetMap.org up and running while we're being deluged by scrapers, we've blocked 320,000+ primarily residential IPv4 addresses in the last 24 hours (+ 100,000 IPv6) involved in scraping.

[chestycougth@mastodon.social]

@osm_tech Thank you. I'm a beginner who has just been doing toy projects and has barely any notion of what web scraping is but I'm very happy to learn that your data can be downloaded

[grechaw@sfba.social]

@zymurgic @osm_tech this kind of abuse has become normal and normalized. It's the AI way. Makes it tough for the legit crawlers out there, too.

[marcel@waldvogel.family]

@HunterZ @osm_tech
My first guess would be some dual-use browser extension. Aka Trojan.

[apnoe_soeren@mastodon.social]

@osm_tech Limit the speed to Modem 14400 speed each IP for a month or so.

[vampirdaddy@chaos.social]

@JonSaenzAgirre @osm_tech
The scrapers are DUMB.
They are not curated, have only basic maintenance, are built to gobble up ANYTHING textual they encounter, without respect, mercy or reason.

Just collect meaningless data.

That’s the nature of the coveted LLMs: just statistics, no understanding, structure or meaning.

And greedy crooks in haste to make quick money just grab everything they can.

The AI bubble needs to pop really soon.

[jkb@gotosocial.jkbockstael.be]

@ClariNerd @osm_tech Because their IP ranges are increasingly being blocked by servers following their harmful scraping habits, AI companies are now releasing "browsers" so they can scrape from residential IPs instead and circumvent blocks. Oh, sorry, I meant "so they can empower users with AI insight in this new era of information".

[clarinerd@mastodon.social]

@jkb @osm_tech brb repeatedly slamming my forehead against my desk for the next five minutes. Then I will reread that and hopefully it will seem less dystopian.

[jonsaenzagirre@mastodon.eus]

@vampirdaddy @osm_tech this seems a reasonable explanation. Quantity of bytes irrespective of sense. Thank you

[felixcremer@fediscience.org]

@osm_tech @utf_7 Why is the first node in OSM somewhere in Italy? I would have expected to find it in some random part of London?

[ondrejzizka@witter.cz]

@osm_tech ‍️

[ondrejzizka@witter.cz]

@osm_tech @michel42 Understood.

Unrelated: Could you please provide me a list of cca 150k random large unsigned integers? I'm testing the xz library and need some test data.

[simon@en.osm.town]

@felixcremer @utf_7 because you are looking at version 43 of the node which has been subject to redaction (licence change), vandalism, and simply buggy software over 20+ years https://www.openstreetmap.org/node/1/history#map=18/1.999999/2.000000

[felixcremer@fediscience.org]

@simon @utf_7 Thanks, yeah that makes sense.

[simon@en.osm.town]

@felixcremer @utf_7 I didn't mention this, but should have: prior to OSM API 0.5 (October 2007) objects were not versioned, the original "node 1" was deleted prior to that date and therefore doesn't actually exist in the current OSM data at all. The current "node 1" is a reuse of the old id IIRC.

[harry_wood@en.osm.town]

@zymurgic The website interface designed for humans is the main issue I believe. See also https://en.osm.town/@osm_tech/115974391032358572
So that's... stupid

I'm not sure who hosts the main Overpass API instance, but I don't think it is the OpenStreetMap Foundation, so (while they probably do have similar challenges) it's not that we're talking about.

[utf_7@mastodon.social]

@simon @felixcremer til something about osm nodes. what distance are 2 neighboring nodes away? or does this vary of the resolution of the area. like on the high seas there are more miles away than in Detroit

[simon@en.osm.town]

@utf_7 @felixcremer the easiest way to calculate this is to use the Haversine distance, see https://en.wikipedia.org/wiki/Haversine_formula

Outside of that nodes are placed where they are deemed necessary to replicate the geometry of the objects. Naturally a rendering on a map can smooth that out if the designer wants to (most don't though).

[jessienab@wetdry.world]

@ryanprior @olbohlen @HunterZ easiest is to just block http 1.1 requests for sites being hammered, since 99% of scrape requests I've seen have been with that protocol.

[algernon@come-from.mad-scientist.club]

@arichtman WTF are Mull doing. Chrome, but no sec-ch-ua.

I'm not having much luck in finding their Android browser... I'm seeing Mullvad VPN, and the browser in alpha for win/mac/linux, but not android. Can you point me in the right direction?

Not going to dive into it now, but I'd like to save it for my records.

[vampirdaddy@chaos.social]

@jay0 @HunterZ @osm_tech

Until recently I mainly fought against residential proxys facilitating DDoS- and crawling-attacks.

Thus I did not have the access threat to internal systems on my radar.

I think that vector is under-reported:

Residential (i.e. software- or library-embedded) proxys on smartphones that are allowed into company networks.