Firefishy

I am part of the team which runs the OpenStreetMap.org infrastructure, we also get a lot of weird and wonderful requests / emails. .mil, .gov, .edu and GeoTLD variants.

I should really do a write-up sometime.

Disclosure: I am part of the mostly volunteer run OpenStreetMap ops team.

Technically we able to block and restrict the scrapers after the initial request from an IP. We've seen 400,000 IPs in the last 24 hours. Each IP only does a few requests. Most are not very good at faking browsers, but they are getting better. (HTTP/1.1 vs HTTP/2, obviously faked headers etc)

The problem has been going on for over a year now. It isn't going away. We need journalists and others to help us push back.

Disclosure: I am part of the OpenStreetMap mostly-volunteer sysadmin team fighting this.

The scrapers try hard to make themselves look like valid browsers, sending requests via residential IP addresses (400,000+ IPs at last count).

I reached out to journalists because despite strong technical measures, the abuse will not go away on its own.

OpenStreetMap Slack https://slack.openstreetmap.us/ was forced to downgraded to the free edition earlier this year for similar reasons.

I help to run the OpenStreetMap infrastructure, and for many years the OSUOSL has generously hosted some of our servers.

Without OSUOSL OpenStreetMap would be more difficult to host and significantly slower to access from North America.

I hope OSUOSL can get the financial support they rightfully deserve.

https://protectli.com/ Good quality devices. Real serial consoles to allow recovery when you make a networking configuration mistake ;-)

Raspberry Pi's "Making a More Resilient File System" document https://pip.raspberrypi.com/categories/685-app-notes-guides-... has instructions on how to configure the eMMC on the CM4 and CM5 to run in pSLC mode. Halving the storage capacity.

OpenStreetMap is now back up and running :-)

The new stack has some unique feature like the vector tiles being minutely updated directly from OSM mapping changes.

There are still issues to fix as it is still only a technical preview.

We don't offer 2x raster tiles because we simply don't have the resources to do it while keeping the tiles minutely updated and open access. We serve around 60k req/sec on a tiny donation backed budget.

The raster tiles are primarily for our OSM mappers to see their map changes rendered quickly to improve the feedback loop.

OP here. The toot was my sarcastic response after having to rate-limit and block another set of abusive scrapers aggressively hitting our website and mapping API. robots.txt be damned.

OpenStreetMap data is free to download. We published minutely on https://planet.openstreetmap.org/ and the data available via AWS S3 + torrent.

If you just starting out, best to start with a smaller regional extract: https://wiki.openstreetmap.org/wiki/Planet.osm

Our data is already publish via torrent, see https://planet.openstreetmap.org Our data, including minutely updates is available via public S3 buckets (EU & US) supported by the AWS Open Data Sponsorship Program.

We publish a live stream of minutely updated OpenStreetMap data in ready do digest form on https://planet.openstreetmap.org/ and S3. Scraping of our data still happens.

Our S3 bucket is thankfully supported by the AWS Open Data Sponsorship Program.

FYI: https://github.com/mchehab/rasdaemon (replaced mcelog) is the daemon for watching for these ECC and other kernel reported "Reliability, Availability and Serviceability" errors.

rasdaemon also attempts to report which physical DIMM / slot triggered the ECC error.

Sub-domain takeover attack. The sub-domain was CNAME'ed to a S3 bucket and the S3 bucket had likely been deleted. The porn purveyor, re-created a new S3 bucket with pr0n.

A scanner that would have caught the vulnerability: https://tech.ovoenergy.com/how-we-prevented-subdomain-takeov...

Or a grey hat scanner for finding sub-domains vulnerable to takeover: https://github.com/m4ll0k/takeover

Wikipedia updates pending ;-)

The vandalism was reverted in 2 hours over 20 days ago. If Mapbox had stayed current with the diffs we publish, the vandalism would have been gone in a blink.

Yes, here is the changelog: https://www.openstreetmap.org/changeset/61555047?node_page=3

Correct. The vandalism was reverted within 2 hours and a data update publish a minute or 2 later. Mapbox import the original vandalism change only today for some reason and there seems to have been a delay in importing the revert.

Pretoria was mostly mapped by a single mapper very early in the project. There wasn't high quality aerial imagery available when he literally cycled every street in Pretoria.

Because the map "looks complete" Pretoria hasn't really attracted many mappers since his departure.

Cape Town has a much more vibrant community of mappers. I've also been able to get crazy high resolution aerial imagery for CT from the municipality which helps: http://coct.aerial.openstreetmap.org.za/

And the Plantronics BackBeat PRO headphones: http://www.plantronics.com/uk/product/backbeat-pro

I hope I am mistaken... It is far from clear as an outsider.

But the 10% or 3% limit respectively is per TAG (handle). HostEurope have multiple tags (HOSTEUROPE, 123-REG, WEBFUSION, WEBFUSION2, DOTUKDOMAINS, ... )

This isn't good news for the .uk domain space.

Nominet (who run the .uk ccTLD) allocate votes proportionally to registrars based on how many domains they have registered. [1]

GoDaddy will now likely control the majority of the votes.

1: http://nominet-prod.s3.amazonaws.com/wp-content/uploads/2015...

It disables some of the network card's hardware offloading. Some have occasional faults, as low as every 4TB of data. The corrupt packets has valid TCP CRC. Commonly first spotted with an ssh error of "Corrupted MAC on input"

Example of an OpenStack vendor guide recommending turning the offloading off: http://clouddoc.stratus.com/1.5.1.0/en-us/Content/Help/P03_S...

Downside is you may have slightly higher CPU usage and/or slightly lower network throughput, I personally have not noticed a significant drop in performance.

Seeing random data corruption over network, try: /sbin/ethtool -K ethX gso off tso off sg off gro off

udev rule something like: SUBSYSTEM=="net", ACTION=="add", ATTRS{vendor}=="0x8086", ATTRS{device}=="0x10c9", ATTRS{subsystem_vendor}=="0x103c", ATTRS{subsystem_device}=="0x323f", RUN+="/sbin/ethtool -K $name gso off tso off sg off gro off"

Review me..

Amazon.co.uk: https://www.amazon.co.uk/HP-ENVY-Printer-Instant-Ready/dp/B0... Amazon.com: https://www.amazon.com/HP-F0V69A-B1H-Wireless-Printing/dp/B0...

And before you say that will never happen read giffgaff (cellphone company) privacy policy: https://www.giffgaff.com/boiler-plate/privacy

Section "3. The information we collect" then "5.iv. To third parties from whom you have chosen to receive marketing information."...

Which means they can give your browsing history, numbers you call etc to any company who you have agreed to receive marketing information from. Eg: Tesco Club Card, that-random-website-you-forgot-to-untick-the-box-on.com etc.

Having the govt require ISPs to collect this data about us will result in ISPs "aggregating" the data and selling it to advertising / marketing firms, insurers or anyone willing to cough up a few £ for your private data.

We'd love for you to get involved and help improve the map :)

Fixed thanks to TomH: http://git.openstreetmap.org/rails.git/commitdiff/2b7529a3f1...