DuckDB-Wasm: Efficient analytical SQL in the browser

Re:DuckDB-Wasm: Efficient analytical SQL in the browser by maxmunzel on Saturday October 30, 2021 @03:43AM (#29046858)

For my bachelor thesis, I used Postgres on a compressed btrfs partition. For my text-heavy dataset, this gave excellent results without compromising on ergonomics.

As the implementation is block-based it is also faster than the naive approach of just zipping your data files.

[ Parent | on HN ]

Re:DuckDB-Wasm: Efficient analytical SQL in the browser by sitkack on Saturday October 30, 2021 @06:42PM (#29052875)

That is really cool, I tried btrfs a couple times and I had weirdness, sorry for being unspecific, not trying to do a drive by hit job on btrfs, but I moved on.

That said, for a specific case like this, I think this could be really compelling, esp going from compressed source material, to a postgres instance running on loopback block device with btrfs on top, could be really amazing for text analysis like you said. You could actually see performance improvements due to this. 4GB/s for PCIe 4, then due to massive core counts and large L3 caches, you could probably boost that to 20GB/s after decompression.

If you can use compression algorithms that allow for regex search in the compression domain, it could be an effective search speed of 10s to hundreds of GB/s.

I am going to have to play with btrfs again.

[ Parent | on HN ]