Re:Monitoring Microservices with Prometheus and Grafana
by saifulwebid on Sunday December 11, 2022 @12:04AM
(#33940296)
How do you “import” existing Prometheus metrics block into Cortex/Mimir using the remote-write, then?
Re:Monitoring Microservices with Prometheus and Grafana
by packetslave on Thursday December 15, 2022 @12:19PM
(#34002690)
For Thanos, you can (mostly) just copy the blocks from the existing s3 bucket into Mimir's bucket. There's some cleanup prep around external labels, but nothing too difficult, and the process is well-documented. I just finished migrating a year of Thanos metrics into Mimir without issue.
It doesn't rely on reading the Prometheus WAL (rather, it uses Prometheus's built-in remote-write functionality). This is both less fragile and has the benefit of being about to run Prometheus in stateless mode (or replace it entirely with something like Grafana Agent, vmagent, etc.
Grafana Labs has done a proof of concept showing Mimir scaling to 1 billion active timeseries. There are also "real" users serving 700+ million metrics in production.