News for hackers. Stuff that matters.

A comment on Two Tier Architectures Are Anachronistic [video].

Re:Two Tier Architectures Are Anachronistic [video] by mike_hearn on Friday November 24, 2023 @12:01PM (#38405924)
It's a tech talk so I watched it and made some notes. It's an argument for using in-process databases when doing data science, rather than an external DB. The speaker pitches DuckDB as a concrete example, which seems to be an in-process DB for Python data frames.

The speaker presents measurements showing how much overhead the wire protocols for various DBs have. MySQL is the best, Postgresql is orders of magnitude worse due to a very inefficient binary format design. The best is still 10x worse than netcat.

Apache Arrow is trying to design a universal protocol for DB access that's more efficient than what's out there currently.

Speaker asserts that scale-out is usually not needed in data analytics, no need to use Spark etc unless you want it on your CV.

Audience member asks "what about multi-user/multi-process access", speaker admits DuckDB basically doesn't do that.

Speaker pitches for using embedded in-proc DBs inside AWS Lambda functions. Not practical to install Oracle RDBMS in something that only runs for 100msec.

A web shell for DuckDB is demonstrated, it uses WASM.

Decentralization is pitched as a reason to avoid 2-tier architecture (separate db engine w/ client protocol).

[ Parent | on HN ]
Re:Two Tier Architectures Are Anachronistic [video] by city_guy_1 on Friday November 24, 2023 @02:49PM (#38407496)
So this is only relevant for personal developer environments?
[ Parent | on HN ]
Re:Two Tier Architectures Are Anachronistic [video] by TN1ck on Friday November 24, 2023 @03:23PM (#38407827)
> Speaker pitches for using embedded in-proc DBs inside AWS Lambda functions. Not practical to install Oracle RDBMS in something that only runs for 100msec.

It's not only unpractical, but hard to get it done. Recently tried to run Postgres in an AWS Lambda to create an anonymized DB dump. It was so painful that I gave up and created an access restricted database to do the anonymization instead. An in-memory mode for Postgres that would be as easy to run as sqlite or duckdb would be so useful for things where one can not replace it with either of them (sql dumps, testing).

[ Parent | on HN ]
Re:Two Tier Architectures Are Anachronistic [video] by wild_egg on Friday November 24, 2023 @03:51PM (#38408170)
You may be interested in this

https://github.com/zonkyio/embedded-postgres-binaries

I've been using this for test runners in Node and Go for a while now and it's been quite painless. Would be nice to have wider language support though

[ Parent | on HN ]
Re:Two Tier Architectures Are Anachronistic [video] by TN1ck on Saturday November 25, 2023 @07:38AM (#38413132)
Thank you, that's indeed super interesting for me (we use go as our backend language).
[ Parent | on HN ]
Re:Two Tier Architectures Are Anachronistic [video] by fulafel on Saturday November 25, 2023 @04:44AM (#38412448)
DuckDB is actually a interesting case because it seems to have some history of memory corruption / segfault problems. The robustness provided by a process boundary is traditionally been valued a lot, though usually for keeping the app from corrupting the db and not the other way around.
[ Parent | on HN ]