pahae

I set up a fairly large Prom-based architecture which I later on migrated to VictoriaMetrics (VM) so I think I can chime in here.

Both Prom and VM are exceptionally stable in my opinion, even on _very_ large scales. There were times when I had a single (Prom, later VM) and not-overly-large instances scrape 2Mio samples/s without any issues. In addition to fairly spiky query loads.

However, if something does go wrong, the single most impactful difference between VM and Prom is simply the difference in startup time. Prometheus with 2TB of metrics takes _forever_ to start up. We're talking up to 2 hours on SSD while VM just... starts.

Remote Write is a viable alternative in Prometheus and its drop-in replacements. I'm not a massive fan of it myself as I feel the pull-based approach is superior overall but still make heavy use of it.

The pushgateway's documentation itself calls out that there are only very limited cirumstances where it makes sense.

I personally only used it in $old_job and only for batch jobs that could not use the node_exporter's textfile collector. I would not use it again and would even advise against it.

I mentioned it in another reply, but https://perses.dev/ is probably the most promising alternative.

Besides that, if you're feeling masochistic you could use Prometheus' console templates or VictoriaMetrics' built-in dashboards.

Though these are all obviously nowhere near as feature rich and capable as Grafana and would only be able to display metrics for the single Prom/VM node they're running on. Might enough for some users.

I only know of https://perses.dev/ but haven't had a look at it for ~half a year. It was very barebones back then but I'm hopeful it can replace Grafana for at least basic dashboarding soon.

I agree. I think OP has made the mistake of using more than just Grafana for dashboards and perhaps user queries.

I operate a fairly large custom VictoriaMetrics-based Observability platform and have learned early on to only use Grafana as opposed to other Grafana products. Part of the stack used to use Mimir's frontend as caching layer but even that died with Mimir v3.0, now that it can't talk to generic Prometheus APIs anymore (vanilla Prom, VictoriaMetrics, promxy etc.). I went back to Cortex for caching.

Such a custom stack is obviously not for everyone and takes much more time, knowledge and effort to deploy than some helm chart but overall I'd say it did save me some headache. At least when compared to the Google-like deprecation culture Grafana seems to have.

I wish the big providers would offer some sort of trial period where you can evaluate models in a _realistic_ setting yourself (i.e cli tools or IDE integrations). I wouldn't even mind strict limits -- just give me two hours or so of usage and I'd already be happy. Seriously.

My use-case is probably pretty far from the usual tasks: I'm currently implementing a full observability platform based on VictoriaMetrics / Victorialogs + Grafana. It's quite elaborate and has practically no overlap with the usual/cloud solutions you find out there. For example, it uses an authenticated query stack: I use the Grafana oauth token to authenticate queries by injecting matchers via prom-label-proxy and forward that to promxy for fan-out to different datasources (using the label filter to only query some datasources). The IaC stuff is also not mainstream as I'm not using any of the big cloud providers, but the provider I use nonetheless has a terraform provider.

As you can imagine, there's probably not much training data for most of this, so quality of the responses varies widely. From my experience so far Claude (Sonnet 4.5 ) does a _much_ better job than GTP-5 (Codex or normal) with the day-to-day task. Stuff like keeping documentation up to date, spotting inconsistencies, helping me find blind spots in the Alerting rules, etc. It also seems to do better working with provided documentation / links.

I've been using Claude for a couple of weeks now but recently switched to codex after my subscription to Claude ran out. I was really curious after reading a lot of good things about it but I gotta say, so far, I'm not impressed. Compared to Claude it gives wrong answers much more frequently (at least in this domain). The results it produces take much more effort to clean up than Claude's. Probably on a level where I could just invest the time myself. Might be that I do not yet know how to correctly prompt GPT but giving both tools the same prompt, Claude does a better job 90% of the time.

Anyway, I guess this is my long-winded way of saying that the quality of responses "off the beaten track" varies widely and is worth testing several models with. Especially if your work is not 70+% of coding. Even then I guess that many benchmarks have seized being useful by now?

Correct. This can be disabled [0] but you need to work around this then. Usually you can "just" use host-networking and manage iptable rules manually. Not pretty but in that case you at least know what's done to the system and iptables.

[0] https://docs.docker.com/engine/network/packet-filtering-fire...

Nebula is fantastic, absolute love it. We use it in production. Cert management can be a bit of a pain on a large scale but there's an excellent Terraform provider [0] that can help. Coupled with the Terraform ansible provider and a little bit of scripting you can automate anything related to cert provisioning and renewal.

[0] https://registry.terraform.io/providers/TelkomIndonesia/nebu...

You should give nebula a try. I've recently switched my private VPN setup from wireguard to nebula and am looking into using it for work. It has some really nice features (for our use case), so ymmv. But so far it's been fantastic and very easy to use.

https://github.com/slackhq/nebula

I'd rather have some physical knob that I can use to turn something off, even if I have to bust out the pliers. Good luck shutting off some touch control that's on the fritz.

The stove at my parents place has some touch control to activate another heating ring(?). Not only is that hard to activate with wet or greasy fingers, but after a couple of years of use it keeps turning itself off now. Thank god that thing fails "closed" and does not (yet?) spontaneously activate effectively doubling the heating power.

I've been using something similar: https://github.com/vincentdoerig/latex-css Seems like it could easily be adapted to match the style.

There are quite a few on ebay if you're desperate for one. Probably won't find the newer 4-NIC models, though.

Ebay or a licensed reseller (if there are any) should be easier anyways. I remember them requiring a VAT ID for EU customers. I ordered a couple for a small business so that was no problem back then but private individuals are pretty much out of luck.

What are you referring to exactly? I do remember having trouble with their official OS until I switched to Armbian on my aged BPI-M1+

Looking at their diagrams it seems that the k8s cluster exists solely to handle their monitoring and logging needs which would be extreme overkill, especially since 18k metrics/samples and 7k logs per second are nothing. Plus you now suddenly need a S3-compatible storage backend for all your logs and metrics. Good thing Ceph comes 'free' with Proxmox, I guess.

Deploying an instance of Prometheus with *every host is also unusual, to say the least and I don't quite understand their comment to that. If you don't like a pull-based architecture (which is a valid point) why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex.