Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json!
What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
I'm at the point where I would pay 5 figures a year for something purely to do better alerting inside or alongside Grafana. Clicking alerts together is a nightmare when I have a ton of identical systems I need to configure. Same for dashboards - the limitations of the current mechanism are too severe.
I'd build my own templating mechanism for it, but I still want the alerts visible in Grafana itself. Zabbix has the power to do all this but with a UX that is not ideal....
You would get all the templating and grouping by labels.
Dashboards can showed Prometheus alerts through annotations on the graphs. So you get a visual feedback what was broken when.
Firing alerts are also a metric on Prometheus, so you can list those (or do other stuff with it).
It's not a UI thing though. More the lower layers to get stuff for Grafana.
Alerts are also integrated tightly in dashboards. Forces alerts to be saved/backedup/imported as single json blob. We want separate management of alerts so they can be defined as code and not in the dashboard blob of json!
What makes me chagrined is because of the above issues we have to use prometheus alert manager instead while our colleagues absolutely LOVE grafana itself! We can't duplicate alerts tens of tens times. We don't want that management nor do we want to teach our colleagues jsonnet/ksonnet to generate it. We also don't want permission problems.
Appreciate the second look and please let me know if you have any additional feedback.
I've literally copied an alert's json format, and then tried to post it back and never got it to work.
Here's an example from my bash history:
> curl -X POST -H "Authorization: Bearer $GRAFANA_API_KEY" -H "accept: application/json" -d @rule.json some_endpoint/api/ruler/grafana/api/v1/rules/test1
I spent a solid day trying to play around with this to get it to work. Because of this the alerts are impossible to code review or store in a git source. Which stinks because Grafana's datasource API's would be amazing to use for alerting. But they're either unusable because anybody can change them or the administrator could bork them at any given point (which has happened before), or just undocumented to the point where they are useless.
That's not even to begin on dealing with the "big blob of json" problem [1] that was clearly important enough to be given an entire spot at GrafanaCon, but even Grafonnet is not supported with Grafana 8. There is apparently some CUE way of doing this, but I can't seem to find any official documentation on that.
Anyways, I've moved back to alertmanager for the time being.
edit: is all of grafana labs downvoting the GP? this is very honest and candid feedback here.
[0]: https://editor.swagger.io/?url=https://raw.githubusercontent...
[1]: https://grafana.com/go/grafanaconline/2021/dashboards-as-cod...
Missing basic functionality like that is a dealbreaker.
I would even pay money for an actual offering that worked.