News for hackers. Stuff that matters.

A comment on Optimizing Large-Scale OpenStreetMap Data with SQLite.

Re:Optimizing Large-Scale OpenStreetMap Data with SQLite by DonnyV on Saturday July 06, 2024 @04:54PM (#40893217)
If you convert those PBFs to Parquet files. You can then use Duckdb to search them with sub-sec response. Plus you get the added bonus of being able to host in an S3 type cloud storage.

https://duckdb.org/docs/extensions/httpfs/s3api

[ Parent | on HN ]
Re:Optimizing Large-Scale OpenStreetMap Data with SQLite by highway900 on Sunday July 07, 2024 @04:01AM (#40895955)
This is the business. I highly recommend this solution. Streaming queries via duckdb or polars will use little to memory and be lightning fast for very large datasets.
[ Parent | on HN ]
Re:Optimizing Large-Scale OpenStreetMap Data with SQLite by jtarchie on Monday July 08, 2024 @11:04AM (#40906065)
That's an interesting take.

I also found [flatgeobuf](https://github.com/flatgeobuf/flatgeobuf), which promises something similar-ish. Perhaps not the same interface for querying, but organizing data in an r-tree format for streaming from a blob store.

[ Parent | on HN ]