> not scalable
Yeah sharding it is a pain. We got its to a million active metrics (ie a metric that was updated in the last 5 mins) on a single instance, with the wisper backend.
I suspect the newer DB based backend are better scalability wise.
> no labels, so you must flatten them into a metric name
I _personally_ don't like labels, I believe that metrics should be one dimensional. I know that puts me at odds with others. I can see the appeal as it allows you too easier group instance metrics into service level ones.
For some reason junior devs / SREs always go nuts putting things into labels, I guess it's the same insecurity and/or brain rot of only-one-class-per-file or three-package deep source hierarchies for a total 1k line project.
The rule of thumb I give them is that if it doesn't make dimensional sense to average across across them, never use a label.
Re:Monitoring Is a Pain
by IsopropylMalbec on Tuesday June 27, 2023 @05:31PM
(#36498942)
If a service which provides metrics supports multiple tenants wouldn't those metrics at the very least need a label to indicate which tenant the metrics are for?
Why doesn’t it make dimensional sense to average across tenants?
Check victoriametrics, they got Graphite query language and ingestion protocol, but on top of their storage.
That thing can scale to a billion of active metrics, not just million.
And a new(?) logging ingestion thing: https://news.ycombinator.com/item?id=36429526 https://docs.victoriametrics.com/VictoriaLogs/ https://github.com/VictoriaMetrics/VictoriaMetrics/releases/... (Apache 2)
- not scalable, even at a moderately sized level; at one work we had graphite workers trying to flush-write the data for dozens of hours on a regular basis, and numerous engineers spent a lot of time trying to optimize that with no good outcome
- no labels, so you must flatten them into a metric name