Postgres and Parquet in the data lake

Re:Postgres and Parquet in the data lake by pella on Wednesday May 04, 2022 @11:02PM (#31268887)

> "Generating Parquet"

It is also useful for moving data from Postgres to BigQuery! ( batch load )

https://cloud.google.com/bigquery/docs/loading-data-cloud-st...

Thanks for the "ogr2ogr" trick! :-)

I hope the next blog post will be about GeoParquet and storing complex geometries in parquet format :-)

https://github.com/opengeospatial/geoparquet

[ Parent | on HN ]

Re:Postgres and Parquet in the data lake by pramsey on Thursday May 05, 2022 @01:43PM (#31276405)

GeoParquet just embeds the geometry as WKB, so getting PostGIS geometry is as simple as going "SELECT geom::geometry FROM my_parq_fdw_table". The spec doesn't yet have anything to say about how to get a good spatial sort or carry out a spatial filter, so it's all pretty ad hoc on that end. Thanks to ogr2ogr, generating a GeoParquet file is now pretty straightforward.

[ Parent | on HN ]

Re:Postgres and Parquet in the data lake by rockostrich on Wednesday May 04, 2022 @11:18PM (#31268988)

Cloud SQL has BigQuery connections that can be leveraged. But yea, this seems like a nice solution if you have a postgres instance outside of Cloud SQL. Another approach would be to write the CDC to a message queue and archive that to parquet.

[ Parent | on HN ]