Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by ttunguz on Tuesday November 15, 2022 @08:52PM
(#33617846)
Yes, that's right.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by mr_toad on Tuesday November 15, 2022 @08:53PM
(#33617854)
> I can query huge parquet datasets in cloud storage without spinning up a VM.
You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by danielmarkbruce on Tuesday November 15, 2022 @09:20PM
(#33618024)
Yeah, I thought the exact same thing.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by danielmarkbruce on Tuesday November 15, 2022 @10:21PM
(#33618410)
>> it already replaced pandas and dask for most of my common data eng tasks
This is interesting - why did you replace pandas/dask with this? What are you getting out of it?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by m_ke on Wednesday November 16, 2022 @05:07PM
(#33631487)
It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc.
I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by danielmarkbruce on Thursday November 17, 2022 @09:53AM
(#33639633)
Gotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by m_ke on Thursday November 17, 2022 @10:45PM
(#33649903)
No because importing into sqlite can be pretty slow. With duckdb you can query parquet files directly.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by danielmarkbruce on Tuesday November 22, 2022 @01:54PM
(#33709520)
Ok, I see that. Many thanks for the insight.
It already replaced pandas and dask for most of my common data eng tasks.