News for hackers. Stuff that matters.
Re:We Built a 19 PiB Logging Platform with ClickHouse and Sa... by rthnbgrredf on Wednesday April 03, 2024 @03:40AM (#39914611)
The use-case of 19 PiB of logging data feels very constructed to me. I worked for smaller and bigger companies and never faced logging into the petabyte range. I'm not saying it's not a thing, FAANG level companies certainly have such needs, but they have their own large scale solutions already. The question remains, besides bragging, who is the average Joe with 19 PiB of logging data you might want to address as potential customer?

What would be useful from my perspective are benchmarks in the more common terabyte range. How much faster is it to query compared to existing cloud offering, what features does e.g. Datadog vs Clickhouse has to analyze the data? In the end the raw data is not much useful if you cannot easily find and extract meaningful data out of it.

Re:We Built a 19 PiB Logging Platform with ClickHouse and Sa... by ryadh on Wednesday April 03, 2024 @05:02AM (#39915001)
We explored a scale similar to what you described in another blog: https://clickhouse.com/blog/cost-predictable-logging-with-cl...

Feature differentiation is actually a pretty interesting topic for o11y. You can do many things with an OLAP store but you need to be aware of the differences with of the shelf solutions, I try to summarize it here: https://clickhouse.com/blog/the-state-of-sql-based-observabi...

I hope this helps! I'd love to hear your opinion about it

Re:We Built a 19 PiB Logging Platform with ClickHouse and Sa... by rthnbgrredf on Wednesday April 03, 2024 @08:33AM (#39916609)
Thanks, this looks indeed much more interesting. I'm definitely looking into this.