Re:New Grafana Loki UI: No LogQL Required
by dotancohen on Friday April 12, 2024 @02:58AM
(#40010057)
OK, I'm starting to see where you're going with this - I also compare incident-affected logs with pre-incident logs, or HTTP requests that I want to debug with similar requests that are known good.
What tools are you using? For me it's often just grep and awk with temp files, maybe a touch of python occasionally.
I pulled this technique from canary analysis and applied it to production outage analysis. In canary, you have a guaranteed random stable population that lets you perform accurate comparisons. Elsewhere, we can try to make that assumption but it might break down. For example, regional holidays can radically alter customer behavior over time or between regions. So it's not perfect but it's often good enough to provide me insights while on call.
And, it requires advanced log queries to perform all these filtering, grouping, counting and scoring functions.