News for hackers. Stuff that matters.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by TexanFeller on Tuesday November 15, 2022 @08:22PM (#33617656)
"My Laptop Is Faster Than Your Cloud" is similar to the things I often say. I've long dismayed at the fact that cloud instances of databases(RDS), build servers, and various other things I need have often been less powerful than the laptop I'm using and at the same time so costly that a few months of rent on them would pay for my laptop.

On the other hand, Postgres can be shockingly fast. With a little care I've been able to make it beat the stuffing out of most other DBs and various big data tools like Spark(running locally, not massively scaled out of course). The article starts out saying it's 80x faster than Postgres. That level of difference from Postgres doesn't impress me, it makes me stop reading and become deeply suspicious that the author either doesn't know anything about how to make PG run well or is cherry picking one of the very few tasks PG doesn't do well or has had a regression in.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by KiranRao0 on Tuesday November 15, 2022 @08:32PM (#33617721)
While this is true that a local computer is often faster than a single cloud instance, I've often noticed code running on cloud instances (dev servers, etc) end up finishing their task more quickly.

It's usually because most tasks I do end up being limited by the network connection to other resources (Postgres DBs, etc). The dev server being in the same datacenter as the resource is a huge advantage in terms of both latency and bandwith.

Again, this is mostly observations from my own workloads, and other people's use cases may be different.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by ttunguz on Tuesday November 15, 2022 @08:53PM (#33617853)
Yes fair point. I see that pattern when running background tasks locally vs on heroku's workers.

In this case, most of the analysis is happening locally without a ton of network use.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by wkdneidbwf on Tuesday November 15, 2022 @09:08PM (#33617935)
exactly. people complaining that an ec2 or rds instance isn’t as fast as their laptop are missing the point.

go ahead and host your production apps on your laptop then. enjoy yourself!

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by wink on Wednesday November 16, 2022 @04:02AM (#33620560)
You're assuming a certain competency in running the cloud setup.

I once had a customer where I've actually used a ~3y old middle of the line laptop (not even a dev one) to prove that our software is not at fault for performance problems because it was running all nodes of the production setup/load better than their cloud setup.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by ttunguz on Tuesday November 15, 2022 @08:27PM (#33617690)
Author here: I agree with you on benchmarks. It's hard to compare different databases well.

MotherDuck is an analytics database optimized for reading with columnar compression. Postgres is more of a transactional/general purpose database. Tuning it well for analytics would surely improve performance.

Most people won't tune a db with custom indices though because it can be hard, so purpose built solutions like this offer value in those cases.

The article isn't purely about performance but also ease of use. MD is an in process database so starting with it is very easy.

Glad we agree on the power within a laptop! It's underappreciated.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Tuesday November 15, 2022 @08:29PM (#33617697)
*motherduck is not a database, it's a company commercializing a great open source project called duckdb.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by carterschonwald on Tuesday November 15, 2022 @08:34PM (#33617731)
this is an important detail I couldn't figure out from the linked materials!
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by ttunguz on Tuesday November 15, 2022 @08:52PM (#33617848)
Good addition. I should have been more clear.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @09:25PM (#33618059)
I don't understand how this is easier than using snowflake.

About the only situation where it seems useful is if I'm using a jupyter notebook and I want to use it to pull data in but it's too much to fit into memory, and sqlite is too slow... which seems like a pretty specific situation to be in? And it's not data that is part of some business process that needs updating frequently?

Or I'm just missing the point entirely? I see another post about huge parquet datasets - but how do I get them? Is everyone at my company comfortable with me moving them onto my machine even if I can?

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Tuesday November 15, 2022 @08:34PM (#33617730)
duckdb is really competing with things like snowflake, where compute is separate from storage. Loading data into postgres and tuning it can be a pain in the ass, with duckdb I can query huge parquet datasets in cloud storage without spinning up a VM.

It already replaced pandas and dask for most of my common data eng tasks.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by ttunguz on Tuesday November 15, 2022 @08:52PM (#33617846)
Yes, that's right.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by mr_toad on Tuesday November 15, 2022 @08:53PM (#33617854)
> I can query huge parquet datasets in cloud storage without spinning up a VM.

You’re querying cloud data locally? Wouldn’t egress costs get expensive quickly?

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @09:20PM (#33618024)
Yeah, I thought the exact same thing.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @10:21PM (#33618410)
>> it already replaced pandas and dask for most of my common data eng tasks

This is interesting - why did you replace pandas/dask with this? What are you getting out of it?

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Wednesday November 16, 2022 @05:07PM (#33631487)
It’s 100x faster, works on larger than memory data and let’s you run SQL over csvs, parquet, arrow tables and etc.

I mostly use it to extract data from a production DB, do basic preprocessing for ML training (which requires joins and filtering that gets ugly in pandas) and typical OLAP queries to compute statistics.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Thursday November 17, 2022 @09:53AM (#33639633)
Gotcha. Out of interest, have you ever done this with sqlite before? I do something similar with sqlite, but speed isn't a factor for me usually because the data is small.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m_ke on Thursday November 17, 2022 @10:45PM (#33649903)
No because importing into sqlite can be pretty slow. With duckdb you can query parquet files directly.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 22, 2022 @01:54PM (#33709520)
Ok, I see that. Many thanks for the insight.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by philliphaydon on Tuesday November 15, 2022 @08:47PM (#33617826)
https://motherduck.com/

Is it this thing? So it’s a non existent service that has 47m in funding? Or am I missing something.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by ren_engineer on Tuesday November 15, 2022 @09:28PM (#33618087)
>or am I missing something

the existence of DuckDB?

https://github.com/duckdb/duckdb

you've got an already popular open source tool being extended and built by the guy who helped make BigQuery and people from Snowflake. VCs are going to be willing to throw money at that

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by Rebelgecko on Tuesday November 15, 2022 @09:38PM (#33618143)
It's super confusing because the graph of github stars vs day shows days that haven't happened yet this month. A lot of the information in the article is presented in a way that makes it hard to see the context.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by giovannibonetti on Thursday November 17, 2022 @08:54PM (#33648817)
That chart shows github stars over the years, from 2019 to the 2022.
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by m463 on Wednesday November 16, 2022 @02:21AM (#33619962)
didn't know about duckdb.

sort of off-topic, I read the "why duckdb" page and found this really nice in this day and age:

"DuckDB has NO external dependencies, neither for compilation nor during run-time.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by nwienert on Wednesday November 16, 2022 @02:49AM (#33620196)
How does that square with the readme?

> For development, DuckDB requires CMake, Python3 and a C++11 compliant compiler.

I guess they mean libraries? But specifying for compilation implies to me they wrote their own compiler or something strange.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by e-clinton on Tuesday November 15, 2022 @09:06PM (#33617916)
Seems like you’ve ducking nailed it. Why was this posted here?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by fbdab103 on Tuesday November 15, 2022 @10:13PM (#33618352)
It's not clear to me what it is. SAAS DuckDB? They want to fork DuckDB into a commercial offering? Stop DuckDB entirely?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by XCSme on Wednesday November 16, 2022 @09:58AM (#33624099)
I still have no idea how all this big sum VC software funding works. Like, what can you even spend $47m on?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by enasterosophes on Tuesday November 15, 2022 @09:03PM (#33617900)
Of course your laptop is faster than the cloud, for things that only require a few cores. (Which is most things, thanks to Amdahl's Law.)

"The cloud" runs on servers with dozens (or even hundreds) of cores, optimized for reliability. The cost of that optimization is each individual core is less fast than a core on a laptop or gaming computer.

The cloud can bring a lot of computing power to gratuitously parallelizable problems, and it allows more effectively sharing and reusing resources, so you can act as a computing utility company instead of every user needing to buy their own on-prem gear.

It doesn't mean the hardware works by magic, which is what is really being expected if you thought optimizations never involve trade-offs.

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @09:03PM (#33617901)
Is this a dimension that matters? Ie while some companies are using snowflake/redshit/bigquery because of scale, many are using it simply because they have their production infra in the cloud, and have a pipeline to move their data into one of those datawarehouse services. It's as much convenience as anything, even if "production infra" is a couple servers and a single database.

How do I get the data on my "laptop" in a consistent (ie, daily updates or whatever) way? How do I share queries and results with other people?

Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by wkdneidbwf on Tuesday November 15, 2022 @09:14PM (#33617974)
you simply expose your database to the internet and then connect to it with user/pass credentials. let me know if you’d like to contract me to set this up for your org!
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by danielmarkbruce on Tuesday November 15, 2022 @09:21PM (#33618033)
This is a joke response?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by whatever1 on Tuesday November 15, 2022 @09:09PM (#33617945)
Aren’t all the cloud instances virtualized? Meaning that at best we start with 80-90% of the bare metal performance?
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by bushbaba on Tuesday November 15, 2022 @09:23PM (#33618050)
More like 99% of bare metal perf for most workloads. https://aws.amazon.com/blogs/hpc/bare-metal-performance-with...
Re:My Laptop Is Faster Than Your Cloud: Announcing MotherDuck by mnkmnk on Wednesday November 16, 2022 @07:51AM (#33622173)
The Motherduck website says “ SERVERLESS DATA ANALYTICS WITH DUCKDB”. Does this mean using lambdas or cloud functions? This looks very similar to https://github.com/cloudfuse-io/buzz-rust