News for hackers. Stuff that matters.
Re:Should You Ditch Spark for DuckDB or Polars? by thenaturalist on Sunday December 15, 2024 @05:04PM (#42426295)
Do you still use file formats at all in your work?

I'm currently thinking of ditching Parquet all together, and going all in DuckDB files.

I don't need concurrent writes, my data would rarely exceed 1TB and if it were, I could still offload to Parquet.

Conceptually I can't see a reason for this not working, but given the novelty of the tech I'm wondering if it'll hold up.

Re:Should You Ditch Spark for DuckDB or Polars? by RobinL on Monday December 16, 2024 @08:50AM (#42430917)
We're still using parquet. So we use the native duckdb format for intermediate processing (during pipeline execution) but the end results are saved out as parquet. This is partly because customers often read the data from other tools (e.g. AWS athena)

I'd be interested in hearing about experiences of using duckdb files though, i can see instances where it could be useful to us