isburmistrov

I agree that specialised DBs outperform a general-purpose OLAP database. The question is - what does outperform mean. In this area queries should not be actually ultra-fast, they should be reasonably fast to be comfortable. And so missing indexes for some attributes would be likely okay. Looking at https://clickhouse.com/blog/storing-log-data-in-clickhouse-f..., they added just bloom filters for columns. Which makes sense, but this is not a full-blown index, and likely it will yield reasonable results. But this all is theoretical, I haven't built such a solution by self (we're working on it now for in-house observability), so likely miss something that can only be discovered on practice.

Btw we use Victoria Metrics now at work. It works good, queries are fast. But we're forced to always think about cardinality, otherwise either performance or cost get hurt. This is okay for the predefined set of metrics & labels and works well, but it doesn't allow having deep explorations.

Thanks for sharing!

> This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.

Really impressive results.

And yet people use ClickHouse quite effectively for this very problem, see the comment here: https://news.ycombinator.com/item?id=39549218

There are also time-series databases out there that are OK with high cardinality: https://questdb.io/blog/2021/06/16/high-cardinality-time-ser...

Thanks for sharing, didn't know about Motif

Does it support "native sampling" described in the article? This is really important to keep the cost low.

Never in the world I would have expected my post to cause the discussion about ODS flaws :D

I think the storage architecture in ClickHouse and Elastic are very different. And I think compression in ClickHouse can be really damn good.

Don't have a good comparison at hands though, but a few random links on the topic:

https://engineeringat.axis.com/schema-changes-clickhouse/ https://clickhouse.com/blog/contentsquare-migration-from-ela... https://clickhouse.com/blog/100x-faster-graphql-hive-migrati... https://blog.zomato.com/building-a-cost-effective-logging-pl...

Haha, I think this term is originated by Honeycomb team actually.

Why I prefer "wide event" over "structured log" as a term because it has this "wide" component that serves for 2 purposes: - it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it

But just a personal preference in the end.

Yes, structured log exactly. Why I prefer "wide event" as a term because it has this "wide" component that serves for 2 purposes:

- it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it

But just a personal preference in the end.

> Those are entirely different orders of magnitude both when it comes to size and how much usefulness you lose. In modern storage backends like Victoriametrics a counter gonna cost you around byte per metric per probe. And as you emit them periodically, that is essentially independent of incoming traffic

I thought this argument was about whether wide events can be used for metrics or metrics is a completely different concept. If we want to emulate metrics in events, we would also make them periodically independently of the traffic. Like emit them once in a while. Pretty much like Prometheus scraping works

I don't think Meta's margins have something to do with this. Companies smaller in scale than Meta also have less data!

And yes, Scuba is in-memory, but it's not the requirement. Check this video out on how Honeycomb implemented their columnar storage: https://vimeo.com/331143124

I think ClickHouse is becoming a default storage for observability nowdays: https://clickhouse.com/use-cases/logging-and-metrics

And there are quite a few solutions on top of it.

A couple of examples that seem to be interesting (however I didn't use them in real life):

https://coroot.com/ https://qryn.metrico.in/#/

Columnar storage stores data very efficiently, too - because it compresses data of a similar nature (columns). Check e.g. ClickHouse on this matter: https://clickhouse.com/docs/en/about-us/distinctive-features, https://clickhouse.com/blog/working-with-time-series-data-an...

So I wouldn't say that events are "expensive" while metrics are "cheap" - both depend on the actual implementation, and events can be cheap too.

And so of course if you have to optimise things, you would need to drop some information you pass to the events, but you would need to do the same for metrics (reduce the number of metrics emitted, reduce the prometheus labels,...).

Sampling can be smart, e.g. based on some field all events have (can be called traceId, haha).

> and "stop sampling" is just a bizarre marketing angle

Wait, where did I mention stopping sampling? :) The opposite: the article is praising the native sampling Scuba has.

I didn't say such tools don't exist. Honeycomb, mentioned in the post, is exactly Scuba fwiw. I said that over-focusing on traces / metrics and "logs" (in the classical understanding) hides the true power of wide events, and they're not being used widely.

Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.