Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by pat2man on Wednesday February 28, 2024 @01:51AM
(#39534857)
The open telemetry collector does just that. https://github.com/open-telemetry/opentelemetry-collector-co...
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by pingiun on Wednesday February 28, 2024 @05:46AM
(#39536359)
seems cool but the top of the page says the thing you suggest is now deprecated
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by nijave on Wednesday February 28, 2024 @07:06AM
(#39536834)
Span Metrics Processor is replaced with the very similar Span Metrics Connector which is still supported
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by paulddraper on Wednesday February 28, 2024 @05:53PM
(#39544548)
The Span Metrics Processor being replaced by the Span Metrics Connector is very, very OpenTelemetry.
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by infogulch on Wednesday February 28, 2024 @02:01AM
(#39534905)
This package implements the parquet idea for Go's slog package: https://github.com/samber/slog-parquet
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by lowbloodsugar on Wednesday February 28, 2024 @01:57AM
(#39534874)
>too big at scale
Bigger than meta? o_0
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by pavlov on Wednesday February 28, 2024 @02:02AM
(#39534906)
The scale isn’t a problem if you also have Meta’s profit margins. But most businesses don’t…
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by rmbyrro on Wednesday February 28, 2024 @03:49AM
(#39535579)
That's what I don't understand about observability services.
They complicate things so much that it's sometimes more expensive to observe than to RUN the infra being observed.
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by elbear on Wednesday February 28, 2024 @04:06AM
(#39535674)
Sounds like an unaligned incentives issue. Or maybe after a certain scale observing becomes cheaper than running?
Re:All you need is Wide Events, not "Metrics, Logs and Traces"
by isburmistrov on Wednesday February 28, 2024 @07:51AM
(#39537199)
I don't think Meta's margins have something to do with this. Companies smaller in scale than Meta also have less data!
And yes, Scuba is in-memory, but it's not the requirement. Check this video out on how Honeycomb implemented their columnar storage: https://vimeo.com/331143124
Why does it have to be an either/or thing? Couldn't you hook up a metrics extractor to the event stream and convert your structured logs to compact metrics in-process before expensive serde/encoding? With this your choice doesn't have to affect the code, just write slogs all the time; if you want structured logs then then output them, but if you only want metrics then switch to the metrics extractor slog handler.
Futher, has nobody tried writing structured logs to parquet files and shipping out 1MB blocks at once? Way less serde/encoding overhead, and column oriented layout compresses like crazy with built-in dictionary and delta encodings.