Re:Querying Parquet with Precision Using DuckDB
by eatonphil on Friday June 25, 2021 @04:35PM
(#27635924)
Sure but what is the scenario where you wouldn't want to use DuckDB and you'd want to use PostgreSQL instead but wouldn't want to load into PostgreSQL? DuckDB is already a SQL engine. The only additional benefit I'd imagine from PostgreSQL would be stuff you get from it being on disk?
But that's why I'm asking, I don't know.
Re:Querying Parquet with Precision Using DuckDB
by taffer on Friday June 25, 2021 @04:51PM
(#27636100)
It depends on your situation: If you already have a Postgres instance then the FDW is probably the easier solution.
Re:Querying Parquet with Precision Using DuckDB
by eatonphil on Friday June 25, 2021 @05:01PM
(#27636227)
Oh yeah that makes sense. I assumed the commenter was just looking for any system to run SQL over Parquet files. If you don't want to use this system that makes it easy, fair enough.
Re:Querying Parquet with Precision Using DuckDB
by taffer on Friday June 25, 2021 @06:18PM
(#27636953)
> If you don't want to use this system that makes it easy, fair enough.
Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community.
- Postgres is a row store optimized for transactional workloads, whereas Parquet is a column-oriented format optimized for analytical workloads.
- Database storage is typically very expensive SSD storage optimized for fast IO and high availability. Parquet files, on the other hand, can be stored in inexpensive object storage such as S3.
- Loading is an additional and possibly unnecessary step