News for hackers. Stuff that matters.
Re:Hydra: Column-Oriented Postgres by gregw2 on Thursday August 24, 2023 @02:51PM (#37253025)
Thanks. Is there a list somewhere of Postgres-based columnar datastores (and which are MPP/clustered vs single/SMP-node)?

* Citus (cstore_fwd) extension

* Azure CosmosDB for PostgreSQL

* GCP AlloyDB for PostgreSQL

* AWS Redshift (but very old Postgres base)

Re:Hydra: Column-Oriented Postgres by hankchinaski on Thursday August 24, 2023 @03:00PM (#37253108)
For sure there are GitHub lists somewhere. I recently had to research for olap Postgres compatible workloads and I know major ones: redshift, clickhouse, hydra
Re:Hydra: Column-Oriented Postgres by wmf on Friday August 25, 2023 @12:44AM (#37258180)
TimescaleDB?
Re:Hydra: Column-Oriented Postgres by claytonjy on Friday August 25, 2023 @10:39AM (#37262233)
I consider Timescale to be more of a hybrid than a "pure" columnar DB. IIRC the latest chunks are row-wise, but it goes columnar for older chunks to compress them. You can do a lot of OLAPish stuff, but it's arguably OLTP-first.
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @01:20AM (#37258373)
Not Postgres-based (but wire- and mostly syntax-compatible): cockroachDB using column families is much like a columnar MPP. Yugabyte is PG-based and MPP but not columnar.
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @01:39AM (#37258484)
> cockroachDB using column families is much like a columnar MPP.

I am wondering why they are saying it is not for OLAP workload..

Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @02:34AM (#37258774)
They don't optimize for it and I suppose the data distribution is primarily aimed at parallel OLTP rather than OLAP. Just wanted to mention that design-wise it is similar but that's indeed not all there is to it. I'd be hesitant to store large volumes of data on a single PG instance; don't see how a single-writer, filesystem-based database is suitable at all for data that is large enough to warrant columnar storage
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @02:46AM (#37258839)
so, what would be your db choice for OLAP?
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @03:33AM (#37259107)
No expert but I'd say you'd rather be looking at bigquery, Redshift, Clickhouse, Snowflake, etc.
Re:Hydra: Column-Oriented Postgres by esafak on Friday August 25, 2023 @02:01PM (#37264968)
You can also go HTAP with TiDB which has TiKV for OLTP and TiFlash (Raft-based columnar replicas) for OLAP.
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @02:11PM (#37265152)
I am more interested in actual OLAP than HTAP, and don't see strong OSS OLAP offering on the market right now, my rants in previous discussion: https://news.ycombinator.com/item?id=36992039

But I should look at TiDB, they looks like interesting and relatively mature project.

Re:Hydra: Column-Oriented Postgres by esafak on Friday August 25, 2023 @02:19PM (#37265255)
https://www.starrocks.io/ is on my shortlist for OLAP
Re:Hydra: Column-Oriented Postgres by riku_iki on Friday August 25, 2023 @02:59PM (#37265784)
its also on my list, but it looks like not easy to try, setup procedure is complicated.
Re:Hydra: Column-Oriented Postgres by refset on Friday August 25, 2023 @06:29AM (#37260062)
The presence and use of column families is only half of the puzzle - it doesn't strictly imply that the execution engine is capable of working in a vectorized columnar style (which is necessary for competitive OLAP).
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @06:49AM (#37260182)
Indeed. As I commented alsewhere this is just about the general design. It is not targeting OLAP in this case (even though I do believe cockroach employs vectorization for reads)
Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @06:52AM (#37260215)
Was unable to edit my previous. It does use vectorization: https://www.cockroachlabs.com/docs/stable/vectorized-executi...
Re:Hydra: Column-Oriented Postgres by refset on Saturday September 02, 2023 @05:32PM (#37365714)
Thanks for sharing that - TIL! These blog posts elaborate with more detail:

https://www.cockroachlabs.com/blog/vectorized-hash-joiner/

https://www.cockroachlabs.com/blog/vectorizing-the-merge-joi...

...it seems the distinction here is that the vectorization is only present in the execution layer and not the storage layer also. I would guess that from a storage perspective, even with column families in play, everything is being streamed out of sorted a LSM engine regardless. So there isn't additionally some highly-tuned buffer pool serving up batches of compressed column files etc.

Re:Hydra: Column-Oriented Postgres by ddorian43 on Friday August 25, 2023 @10:44AM (#37262295)
Note column families has nothing to do with columnar.

Another example is cassandra is not column oriented.

Re:Hydra: Column-Oriented Postgres by hazaskull on Friday August 25, 2023 @11:31AM (#37262847)
Thank you for the correction. Indeed it is not entirely the same thing. Though I'd expect that at least the benefit of not having to read columns that aren't in the family would still help (haven't tried in earnest). I suppose compression is not an option though.
Re:Hydra: Column-Oriented Postgres by mslot on Friday August 25, 2023 @04:24AM (#37259376)
Note that cstore_fdw ≈ Citus columnar (modernized using table access method) ≈ Azure Cosmos DB for PostgreSQL (managed Citus service) ≈ Hydra (a Citus columnar fork)

Fwiw, the main use case we had in mind when developing the Citus columnar table access method was compression of old data in a time-partitioned table. Citus is commonly used for real-time analytics on time series data. That involves more materialization than typical OLAP reporting use cases. However, you might still want to keep the source data in the database for ad-hoc queries or future materialization and it's preferable to keep the source data in compressed form.

Redshift is specifically architected for ad-hoc OLAP queries. AlloyDB I'm not sure.