So what I want is something like Splunk search for traces. I want to be able to write 12-shareable lines of custom-query and immediately get a pretty graph of multidimensional data "What % of users who just registered this month got a 500 more than half the time on this specific path?" and I want it as a pretty graph over time
Re:Monitoring Is a Pain
by phillipcarter on Tuesday June 27, 2023 @05:36PM
(#36499003)
That's precisely what tools like Honeycomb and Lightstep offer. But most vendors don't have a traces-first approach to querying data.
Really? I haven't used honeycomb specifically, but you're saying it supports an arbitrarily complex query depth, with any number of joins essentially?
I kinda hacked x-ray onto things, but it does give you an overview aggregate for each node
I'm not sure if I've really found it useful though. Im sure if I didn't know the service, itd be great, but I already know where to look for whatever problem
I think PromScale looked promising for some of those problems but it's also a deprecated product.
I think the state of the art is/will be ingesting traces into Iceberg for some form of analysis with whatever execution engine you want (Trino/Spark/etc...). A lot of places are doing this.
OpenTelemetry is clearly the future of monitoring, probably with a lot of lifting from Prometheus and other things.