News for hackers. Stuff that matters.
Re:DuckDB Isn't Just Fast by koromak on Tuesday June 11, 2024 @09:39AM (#40646060)
I very, very nearly migrated to a full Duckdb solution for customer-facing historical stock data. It would have been magical, and ridiculously, absurdly, ungodly fast. But the cloud costs ended up being close to a managed analytics solution, with significantly more moving parts (on our end). But I think thats just our use case, going forward I'd look at duckdb as an option for any large-scale datasets.

Using ECS/EKS containers reading from a segmented dataset in EFS is a really solid solution, you can get sub second performance over 6 billion rows / 10000 columns with proper management and reasonably restrictive queries.

Another option is to just deploy a couple huge EC2 instances that can fully fit the dataset. Costs here were about the same, but with a little more pain in server management. But the speed man, its just unbelievable.

Re:DuckDB Isn't Just Fast by mritchie712 on Tuesday June 11, 2024 @11:56AM (#40647844)
Did you check the cost to run it on Motherduck (https://motherduck.com/)?
Re:DuckDB Isn't Just Fast by koromak on Tuesday June 11, 2024 @09:41AM (#40646078)
To note, we migrated from Redshfit, which had 7-30 second performance. Our current managed solution is something like 1 - 5. Duckdb just smashes everything else, at least on our data.
Re:DuckDB Isn't Just Fast by vgt on Tuesday June 11, 2024 @06:56PM (#40652625)
Co-founder and head of produck at MotherDuck here - would love to chat. We're running DuckDB in a serverless fashion, so you're only paying for what you consume.

Feel free to reach out to [email protected].

Re:DuckDB Isn't Just Fast by panyang on Wednesday June 12, 2024 @02:47PM (#40661512)
What do you mean by segmented dataset in EFS?