News for hackers. Stuff that matters.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Tuesday November 15, 2022 @08:34PM (#33617730)
duckdb is really competing with things like snowflake, where compute is separate from storage. Loading data into postgres and tuning it can be a pain in the ass, with duckdb I can query huge parquet datasets in cloud storage without spinning up a VM.

It already replaced pandas and dask for most of my common data eng tasks.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by ttunguz on Tuesday November 15, 2022 @08:52PM (#33617846)
Yes, that's right.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by mr_toad on Tuesday November 15, 2022 @08:53PM (#33617854)
> I can query huge parquet datasets in cloud storage without spinning up a VM.

You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @09:20PM (#33618024)
Yeah, I thought the exact same thing.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @10:21PM (#33618410)
>> it already replaced pandas and dask for most of my common data eng tasks

This is interesting - why did you replace pandas/dask with this? What are you getting out of it?

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Wednesday November 16, 2022 @05:07PM (#33631487)
It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc.

I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Thursday November 17, 2022 @09:53AM (#33639633)
Gotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Thursday November 17, 2022 @10:45PM (#33649903)
No because importing into sqlite can be pretty slow. With duckdb you can query parquet files directly.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 22, 2022 @01:54PM (#33709520)
Ok, I see that. Many thanks for the insight.