News for hackers. Stuff that matters.
Re:A Practitioner's Guide to Wide Events by valyala on Tuesday December 24, 2024 @05:17PM (#42505338)
Wide events is a great concept for observability space! This a superset of structured logs and traces. Wide events is basically structured logs, where every log entry contains hundreds of fields with various properties of the log entry. This allows slicing and dicing the collected events by arbitrary subsets of thier fields. This opens an infinite possibilities to obtain useful analytics from the collected events.

Wide events can be stored in traditional databases. But this approach has a few drawbacks:

- Every wide event can have different sets of fields. Such fields cannot be mapped to the classical relational table columns, since the full set of potential fields, which can be seen in wide events, isn't known beforehand.

- The number of fields in wide events is usually quite big - from tens to a few hundreds. If we are going to store them in a traditional relational table, this table will end up with hundreds of columns. Such tables aren't processed efficiently by traditional databases.

- Typical queries over wide events usually refer only a few fields out of hundreds of available fields. Traditional databases usually store every row in a table as a contiguous chunk of data with all the values for all the fields of the row (aka row-based storage). Such a scheme is very inefficient when the query needs to process only a few fields out of hundreds of available fields, since the database needs to read all the hundreds fields per each row and then extract the needed few fields.

It is much better to use analytical databases such as ClickHouse for storing and processing of big volumes of wide events. Such databases usually store values per every field in contiguous data chunks (aka column-oriented storage). This allows reading and processing only the needed few fields mentioned in the query, while skipping the rest of hundreds fields. This also allows efficiently compressing field values, which reduces storage space usage and improves performance for queries limited by disk read speed.

Analytical databases don't resolve the first issue mentioned above, since they usually need creating a table with the pre-defined columns before storing wide events into it. This means that you cannot store wide events with arbitrary sets of fields, which can be unknown before creating the table.

I'm working on a specialized open-source database for wide events, which resolves all the issues mentioned above. It doesn't need creating any table schemas before starting ingesting wide events with arbitrary sets of fields (e.g. it is schemaless). It automatically creates the needed columns for all the fields it sees during data ingestion. It uses column-oriented storage, so it provides query performance comparable to analytical databases. The name of this database is VictoriaLogs. Strange name for the database specialized for efficient processing of wide events :) This is because initially it was designed for storing logs - both plaintext and structured. Later it has been appeared that it's architecture ideally fits wide events. Check it out - https://docs.victoriametrics.com/victorialogs/

Re:A Practitioner's Guide to Wide Events by bonobocop on Tuesday December 24, 2024 @05:44PM (#42505487)
Thoughts on stuff like ClickHouse with JSON column support? Less upfront knowledge of columns needed.
Re:A Practitioner's Guide to Wide Events by hipadev23 on Tuesday December 24, 2024 @11:36PM (#42506848)
Clickhouse's revised JSON type is still quite new (in beta currently), but I'm hopeful for it. Their first attempt fell apart if the schema changed.

[1] https://clickhouse.com/blog/a-new-powerful-json-data-type-fo...

Re:A Practitioner's Guide to Wide Events by brunoqc on Wednesday December 25, 2024 @12:06AM (#42506981)
ClickHouse is open core too. If you care about that.
Re:A Practitioner's Guide to Wide Events by valyala on Wednesday December 25, 2024 @01:44AM (#42507328)
JSON column type in ClickHouse [1] looks promising, since it allows storing wide events with arbitrary sets of fields. This feature is still in beta. Let's see how it will evolve.

[1] https://clickhouse.com/docs/en/sql-reference/data-types/newj...

Re:A Practitioner's Guide to Wide Events by maxmcd on Wednesday December 25, 2024 @04:00AM (#42507781)
It is a great step, but in my testing with the new JSON type if you use beyond 255 unique json locations/types (255 max_dynamic_types in their config) you will fall back to much worse performance for certain queries and aggregations. This is quite easy to hit with some of the suggestions in this blog post, especially if you are designing for multi-tenant use.

For this clickhouse wide event lib I'm working on (not worth anyones time atm) I am still using this schema https://www.val.town/v/maxm/wideLib#L34-39 (which is from a Boris Tane talk https://youtu.be/00gW8txIP5g?t=801) for good multi-tenant performance.

I hope clickhouse performance here can still be vastly improved, but I think it is a little awkward to get optimal performance with wide events today.

Re:A Practitioner's Guide to Wide Events by bonobocop on Wednesday December 25, 2024 @08:05AM (#42508606)
A small question on the schema, I noticed that you have only “_now” as the Order By (so should just use that for the primary key). Do you expect any cross tenant queries?

Just my feeling would be that I’d add the tenant ID before the timestamp as it should filter the parts more effectively

Re:A Practitioner's Guide to Wide Events by maxmcd on Wednesday December 25, 2024 @08:52AM (#42508771)
Yes, I think you are correct. In the video Boris/Baselime uses (_tenantId, _traceId, _timestamp). Will update that :)
Re:A Practitioner's Guide to Wide Events by oulipo on Wednesday December 25, 2024 @06:01AM (#42508182)
How is that a "superset" ? From what I gather, it's... just a "JSON-formatted log"? They just decide to put as much data in it as they can and decide that it should be called a "wide event", but it makes no sense... it's just a regular JSON-formatted log, with all the data inside, nothing new?
Re:A Practitioner's Guide to Wide Events by brunoqc on Wednesday December 25, 2024 @06:40PM (#42511994)
Any fully open source tool that can store "wide events"?
Re:A Practitioner's Guide to Wide Events by valyala on Wednesday December 25, 2024 @07:04PM (#42512109)
VictoriaLogs - it is fully open source released under Apache2 license, and it can efficiently store and query these "plain old JSONs" with hundreds of fields (aka wide events) - https://docs.victoriametrics.com/victorialogs/keyconcepts/#d...
Re:A Practitioner's Guide to Wide Events by brunoqc on Wednesday December 25, 2024 @07:16PM (#42512175)
Thanks. Yeah, I was planning to try it but I was wondering since the parent's comment was saying it's just a "JSON-formatted log".
Re:A Practitioner's Guide to Wide Events by oulipo on Thursday December 26, 2024 @05:09AM (#42514257)
that's what I infer from their description? They just seem to have slapped a fancy name and say: just add as many data point to your JSON as you can, and call that a "wide event", but I don't see how it's not "just" a JSON-formatted log?
Re:A Practitioner's Guide to Wide Events by valyala on Thursday December 26, 2024 @01:47PM (#42516913)
Wide events are plain old structured logs with many fields. They can be represented as JSON. They also can be represented as logfmt, protobuf or any other format suitable for representing a set of (field=value) pairs.

The novelty of wide events is that it is recommended to:

- emit an event (leg entry) once per every processed request. Previously it was OK to emit many logs per request. This could complicate degugging and analyzing such logs.

- don't afraid to add fields to the event if these fields can help debugging and/or analyzing the logs. Previously it was recommended artificially limiting the number of log fields to some small value. This could prevent from debugging and analyzing such logs in the future.

Re:A Practitioner's Guide to Wide Events by brunoqc on Thursday December 26, 2024 @10:58PM (#42519823)
Can you emit only 1 event per request when you use the open telemetry collector are you emit a bunch of spans? I think I saw that Victoria logs support the otel collector.
Re:A Practitioner's Guide to Wide Events by valyala on Friday December 27, 2024 @06:52PM (#42527204)
You can emit as many events per request as you wish, and store all of them to VictoriaLogs. But it is recommended emitting a single "master" event with all the context and useful information about the request in separate fields, which then can help debugging and analyzing the processed requests.
Re:A Practitioner's Guide to Wide Events by oulipo on Friday December 27, 2024 @06:36AM (#42521435)
One event per request seems okay in some monitoring scenarios, but in debugging scenarios you might want to instrument many functions
Re:A Practitioner's Guide to Wide Events by masterj on Friday December 27, 2024 @04:50PM (#42526373)
One main event per request with a lot of context does not mean that’s the only data you emit. You can emit “normal” log lines too, or use granular log levels, etc.

However for understanding how your system or your users are behaving, querying the wide or “main” events will be far better as entry points for exploration.

Re:A Practitioner's Guide to Wide Events by masterj on Friday December 27, 2024 @04:46PM (#42526334)
It is “just” a JSON-formatted log (or any other format really, just a set of keys and values).

However the practice of collecting a lot of context per-transaction / unit-of-work and emitting that as one piece of data, storing it in a place that can quickly query across these and visualize is not very common across most orgs and teams. Feedback on this article has been a mix of “I’ve never heard of this before” and “we’ve been doing this for a decade, didn’t know anyone had a name for it” with not a lot of in-between.

It’s not a new idea, which I call out in the intro. It’s not even a very fancy idea. It is really, really helpful if implemented though. Modern OLAP column stores help a lot here too since they make this type of exploration cheap and quick.