Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by m_ke on Wednesday November 16, 2022 @05:07PM
(#33631487)
It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc.
I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by danielmarkbruce on Thursday November 17, 2022 @09:53AM
(#33639633)
Gotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by m_ke on Thursday November 17, 2022 @10:45PM
(#33649903)
No because importing into sqlite can be pretty slow. With duckdb you can query parquet files directly.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck
by danielmarkbruce on Tuesday November 22, 2022 @01:54PM
(#33709520)
Ok, I see that. Many thanks for the insight.
This is interesting - why did you replace pandas/dask with this? What are you getting out of it?