News for hackers. Stuff that matters.
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @01:20AM (#37258373)
Not Postgres-based (but wire- and mostly syntax-compatible): cockroachDB using column families is much like a columnar MPP. Yugabyte is PG-based and MPP but not columnar.
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @01:39AM (#37258484)
> cockroachDB using column families is much like a columnar MPP.

I am wondering why they are saying it is not for OLAP workload..

Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @02:34AM (#37258774)
They don't optimize for it and I suppose the data distribution is primarily aimed at parallel OLTP rather than OLAP. Just wanted to mention that design-wise it is similar but that's indeed not all there is to it. I'd be hesitant to store large volumes of data on a single PG instance; don't see how a single-writer, filesystem-based database is suitable at all for data that is large enough to warrant columnar storage
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @02:46AM (#37258839)
so, what would be your db choice for OLAP?
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @03:33AM (#37259107)
No expert but I'd say you'd rather be looking at bigquery, Redshift, Clickhouse, Snowflake, etc.
Re:Hydra: Column-Oriented Postgres by esafak on Friday August 25, 2023 @02:01PM (#37264968)
You can also go HTAP with TiDB which has TiKV for OLTP and TiFlash (Raft-based columnar replicas) for OLAP.
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @02:11PM (#37265152)
I am more interested in actual OLAP than HTAP, and don't see strong OSS OLAP offering on the market right now, my rants in previous discussion: https://news.ycombinator.com/item?id=36992039

But I should look at TiDB, they looks like interesting and relatively mature project.

Re:Hydra: Column-Oriented Postgres by esafak on Friday August 25, 2023 @02:19PM (#37265255)
https://www.starrocks.io/ is on my shortlist for OLAP
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @02:59PM (#37265784)
its also on my list, but it looks like not easy to try, setup procedure is complicated.
Re:Hydra: Column-Oriented Postgres by refset on Friday August 25, 2023 @06:29AM (#37260062)
The presence and use of column families is only half of the puzzle - it doesn't strictly imply that the execution engine is capable of working in a vectorized columnar style (which is necessary for competitive OLAP).
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @06:49AM (#37260182)
Indeed. As I commented alsewhere this is just about the general design. It is not targeting OLAP in this case (even though I do believe cockroach employs vectorization for reads)
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @06:52AM (#37260215)
Was unable to edit my previous. It does use vectorization: https://www.cockroachlabs.com/docs/stable/vectorized-executi...
Re:Hydra: Column-Oriented Postgres by refset on Saturday September 02, 2023 @05:32PM (#37365714)
Thanks for sharing that - TIL! These blog posts elaborate with more detail:

https://www.cockroachlabs.com/blog/vectorized-hash-joiner/

https://www.cockroachlabs.com/blog/vectorizing-the-merge-joi...

...it seems the distinction here is that the vectorization is only present in the execution layer and not the storage layer also. I would guess that from a storage perspective, even with column families in play, everything is being streamed out of sorted a LSM engine regardless. So there isn't additionally some highly-tuned buffer pool serving up batches of compressed column files etc.

Re:Hydra: Column-Oriented Postgres by ddorian43 on Friday August 25, 2023 @10:44AM (#37262295)
Note column families has nothing to do with columnar.

Another example is cassandra is not column oriented.

Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @11:31AM (#37262847)
Thank you for the correction. Indeed it is not entirely the same thing. Though I'd expect that at least the benefit of not having to read columns that aren't in the family would still help (haven't tried in earnest). I suppose compression is not an option though.