Re:Lesser known features of ClickHouse
by polskibus on Tuesday May 31, 2022 @03:08PM
(#31573545)
How’s that? How does clickhouse replace ETLs? How does it alleviate data quality problems? It is just a data warehouse , very fast but with limited data modelling and permissioning compared to MS OLAP or tabular models.
[ Parent
| on HN ]
Re:Lesser known features of ClickHouse
by benjaminwootton on Tuesday May 31, 2022 @04:12PM
(#31574348)
Im not really disagreeing with him, but you can use DBT to materialise 10 daisy chained views, or hit Clickhouse with a query and allow it to crunch data.
I am a proponent of modern data stack but it’s always at the back of my mind that an OLAP database on a big box negates the need for a lot of it.
Sure it does, but you have to feed it somehow and maintain it so that the data there is of good quality and makes business sense. Afaik, clickhouse is not great for multitable joins required by good data warehousing practices like star and snowflake schema.
Re:Lesser known features of ClickHouse
by gauravphoenix on Tuesday May 31, 2022 @06:15PM
(#31575590)
CH is actually very good if you can model the dimensional data as dictionary. Also CH can use external store like PSQL as dictionary. But yes, if your fact table data size is as big as dimensional data, CH or any other such system will be quite slow.
Re:Lesser known features of ClickHouse
by datalopers on Tuesday May 31, 2022 @04:57PM
(#31574859)
ETL: Clickhouse can probably directly read any format or data source you have.
Quality: A diligent human like anywhere.
Data Modeling: I disagree, what do you think it is missing? I’ve never encountered anything I couldn’t achieve, and the Clickhouse specific aggregates, lambda support, and array functionality really simplifies many problems.
Permissioning: It has very granular permissions.
Re:Lesser known features of ClickHouse
by gauravphoenix on Tuesday May 31, 2022 @06:12PM
(#31575571)
If you use the "Object" data type (json), you can have a schema-less datastore and that can alleviate the need of complex ETL pipelines. But yes, if you need to decorate the data before ingestion, you need some ETL pipeline. In other words, if the ETL pipeline is just storing the data in columnar format by reading some JSON fields, you don't need that anymore with the new CH json data type