Re:DuckDB over Pandas/Polars
by swasheck on Tuesday November 05, 2024 @09:50PM
(#42056935)
we use partitioned parquet files in s3. we use a csv in the bucket root to track the files. i’m sure there’s a better way but for now the 2tb of data are stored cheaply and we get fast reads by only reading the partitions we need to read.
[ Parent
| on HN ]
I'm curious how much simpler to build, manage, and run vs cost it would be to simply running a database on a large vultr/DO instance and paying for 2tb of storage?
I feel like you'd get away with the whole thing for around $500/mo depending on how much compute was needed?
You just need to try it once to see the issue. Merely loading this amount of data onto a Postgres db will be hell.
well that's not the infrastructure we have. we are primarily an aws shop so we use the resources available to us in the context of our infrastructure decisions. it would be a hard sell to buy something outside of that ecosystem.
I understand that's the infrastructure you have. But that's more describing vendor lock-in haha.
Most of my work is with clients that don't have any set infrastructure yet, so was curious if anyone had any anecdotes.