News for hackers. Stuff that matters.
Re:How to Efficiently Choose the Right Database for Your App... by mb7733 on Sunday February 28, 2021 @11:38AM (#26294527)
In common usage does OLAP automatically imply data that is too large for a single node to handle? I'm wondering if a lot of OLAP workloads couldn't be handled with some parquet files (or some other column based storage)?

I realize parquet is used by the hadoop/spark ecosystem, but do you really need those systems? I'm thinking that a lot of companies reach for a hadoop cluster when some parquet files in a regular for system would be much simpler. I've done things like this and in my experience it works quite well. But only for personal projects.

Re:How to Efficiently Choose the Right Database for Your App... by glogla on Sunday February 28, 2021 @12:54PM (#26295284)
No, of course you're right, I'm just bit too comfortable in my own domain.

For smaller data (and here that would mean few hundreds GB which is pretty huge by normal standards) OLTP databases would cover you pretty well. Oracle has some fancy bitmap indexing, MS SQL even has columnar tables, and pretty much anything has partitioning.

Parquet is really cool - especially there's like 30 x ratio between "generic oracle table" and gzip compressed parquet file so scan time are really in a different world.

But by itself parquet doesn't solve a whole lot - where are the files stored and what scans them? What happens if someone is updating the files while someone else is reading them?

Re:How to Efficiently Choose the Right Database for Your App... by mb7733 on Friday March 05, 2021 @11:08AM (#26358381)
Yeah the storage size was a real clincher for me, as much as I appreciate a SQL database.

Re: concurrent reading and writing, for my use case the files are immutable so that isn't a concern. But I agree, I don't think pure parquet is a good fit there.