News for hackers. Stuff that matters.

A comment on Switch to OpenStreetMap. Parent comment

Re:Switch to OpenStreetMap by jmnicolas on Thursday December 12, 2019 @09:42AM (#21772166)
I tried to build my own OSM server a couple months back. I quickly abandoned when I discovered that the amount of processing to import their world map data file in Postgres would require several months (desktop Core i7 3770k / 16 GB ram but a 7200 rpm hard drive since there wasn't enough space on my SSD).

For a world map server you would need a beefy machine to make it less painful, something like 64 GB ram, multi nvme ssd and countless cores.

I wish they offered a Postgres dump of it.

[ Parent | on HN ]
Re:Switch to OpenStreetMap by nathancahill on Thursday December 12, 2019 @09:49AM (#21772228)
Hmm, I run a handleful of OSM servers with similar specs and it takes a little a day or 2 to import the world to a HDD. Here's some benchmarks. It definitely helps to have a beefy server: https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks

A Postgres dump would be massive.

[ Parent | on HN ]
Re:Switch to OpenStreetMap by growlist on Thursday December 12, 2019 @11:23AM (#21773219)
> A Postgres dump would be massive.

Indeed - it inflates to hundreds of GB during my import process.

[ Parent | on HN ]
Re:Switch to OpenStreetMap by growlist on Thursday December 12, 2019 @11:22AM (#21773215)
> the amount of processing to import their world map data file in Postgres would require several months

I'd be surprised if this were indeed the case, even on spinning disks. Did you use the pbf export? I can load it in two or three days using osm2pgsql on my home desktop (albeit with SSDs). On the £10k+ work server it loads in about 24 hours.

The issue with providing a Postgres dump is that people often want different things from the output data, and the choices made during the import might rule out certain uses.

[ Parent | on HN ]
Re:Switch to OpenStreetMap by jmnicolas on Monday December 16, 2019 @10:51AM (#21803914)
osm2pgsql ran a few hours for the first phase, then I let it run a couple days on the second phase (there are 3 phases IIRC).

I then calculated how long it would take to finish the second phase according to how much it had accomplished in those few days and I calculated that I would have to let my PC on until December 15 (it was back in September or October I think).

[ Parent | on HN ]
Re:Switch to OpenStreetMap by samcrawford on Thursday December 12, 2019 @11:12AM (#21773092)
FWIW, I have been doing exactly this in the past few months, importing planet data using osm2pgsql.

It takes about 18 hours to import the whole planet onto a E5-1650v3 server, with 256GB RAM and a 1TB SSD. 128GB RAM would certainly have sufficed. The number of cores doesn't actually matter much. I've never seen it use more than 8 cores, even when I ask it to use far more.

[ Parent | on HN ]