Systems Monitoring with Prometheus and Grafana

Re:Systems Monitoring with Prometheus and Grafana by bacondude3 on Tuesday August 11, 2020 @09:57PM (#24127906)

What's a good alternative to Prometheus when pulling stats is impractical? Say I want to monitor a personal laptop like I would a server. It will change networks and IP addresses, so pulling would be impractical to configure, whereas the laptop could easily(?) push its stats to a remote server.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by MetalMatze on Wednesday August 12, 2020 @04:09AM (#24129644)

Prometheus supports writing (replicating) data to a remote endpoint on a per scrape basis with a protocol called remote-write. You can pretty easily set that up on any Prometheus instance. There are quite some implementations to receive those remote-write requests: https://prometheus.io/docs/operating/integrations/#remote-en...

You're probably exactly looking for something like that. In fact, I've given a talk about a similar scenario at the KubeCon San Diego: https://www.youtube.com/watch?v=FrcfxkbJH20

Disclosure: I work on Thanos and Thanos Receiver which implements that protocol.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by bacondude3 on Wednesday August 12, 2020 @11:38AM (#24133126)

Oh, I had no idea about that. Thanks!

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by cassianoleal on Wednesday August 12, 2020 @04:48AM (#24129839)

I've been using ZeroTier [0] so whenever my laptop is online, it's also accessible by Prometheus so it can be scraped. I run ZeroTier in a container on my router, and on each "road warrior" that needs LAN access (laptop, phone...).

The caveat is that I have no metrics when the laptop is offline but that doesn't happen very often anyway.

[0] https://www.zerotier.com/

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by bacondude3 on Wednesday August 12, 2020 @11:37AM (#24133108)

That looks like a really cool solution. Thanks for the suggestion!

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by valyala on Wednesday August 12, 2020 @12:41PM (#24133831)

You can run lightweight vmagent [1] on your laptop. The vmagent collects all the metrics from your laptop and then sends them to remote storage when connection to the remote storage is available. It can send data to any remote storage system that support Prometheus remote_write protocol [2].

[1] https://victoriametrics.github.io/vmagent.html

[2] https://prometheus.io/docs/operating/integrations/#remote-en...

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by bacondude3 on Wednesday August 12, 2020 @06:26PM (#24137783)

That looks like a really promising solution. Thanks!

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by bvrmn on Tuesday August 11, 2020 @10:57PM (#24128198)

You can use push approach. Like graphite. Grafana has excellent support of graphite queries. Also graphite has low-footprint implementation https://github.com/baverman/hisser

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by molecule on Wednesday August 12, 2020 @12:02AM (#24128492)

Telegraf + InfluxDB?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by abhishekjha on Wednesday August 12, 2020 @01:15AM (#24128794)

This is my setup on all my raspberrypis. I have not be been able to figure out how to monitor a cluster though. I saw that grafana free tier doesn’t allow a cluster of servers getting monitored. I have telegraf + influxdb + grafana installed on all my servers.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by michaelmcdonald on Wednesday August 12, 2020 @07:21AM (#24130843)

Could you expand on what you mean by:

> grafana free tier doesn’t allow a cluster of servers getting monitored.

Is there a particular aspect of the cluster you're missing? Is it that you don't want individual server metrics?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by abhishekjha on Wednesday August 12, 2020 @09:20AM (#24131772)

I have telegraf + influxdb + grafana-server installed on each of my Rpis giving me multiple dashboards. I want only one grafana-server dashboard where all the telegraf metrics could be seen.

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by valyala on Wednesday August 12, 2020 @12:42PM (#24133841)

vmagent + victoriametrics :)

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by cortesoft on Tuesday August 11, 2020 @10:47PM (#24128125)

Use a push gateway?

[ Parent | on HN ]

Re:Systems Monitoring with Prometheus and Grafana by ncrmro on Wednesday August 12, 2020 @07:11PM (#24138165)

If you use wireguard you can tell Prometheus to scrape your ip on the wireguard network

[ Parent | on HN ]