News for hackers. Stuff that matters.
Re:Faster PostgresSQL to BigQuery by vidarh on Tuesday January 10, 2023 @07:12AM (#34323607)
Back when I was working with shapefiles, it was the type of things that tended to be far more convenient to process in-process using something like GDAL [1] (which can operate directly on an in memory copy, gzip files, sqlite databases and far more) and query it with GDAL's SQL support, especially when build with Spatialite [2] rather than loading it into a separate database. It'd have been interesting if the author had talked about what's stopping him from that approach given he's clearly aware of GDAL and given that 130M records and a few tens of GB isn't a particularly big GIS dataset.

[1] https://gdal.org

[2] https://www.gaia-gis.it/fossil/libspatialite/index

Re:Faster PostgresSQL to BigQuery by marklit on Tuesday January 10, 2023 @07:22AM (#34323718)
I'll take a look into it.

Last year I found ClickHouse did H3 enrichment much faster than PG or BQ so it has become a go-between for larger workloads with simple enrichment requirements.

The dataset in the post is an example anyone can download. I handle 30B+ records at work.

Re:Faster PostgresSQL to BigQuery by qoega on Tuesday January 10, 2023 @11:45AM (#34327567)
>It would also be good to see GEOS, GDAL and PROJ integrated into ClickHouse.

You can just add feature requests like this to ClickHouse issue tracker. https://github.com/ClickHouse/ClickHouse/issues/45129

Re:Faster PostgresSQL to BigQuery by marklit on Tuesday January 10, 2023 @12:31PM (#34328374)