News for hackers. Stuff that matters.
Re:Show HN: Homelab Monitoring Setup with Grafana by danwee on Wednesday June 07, 2023 @10:57AM (#36227598)
> It's easy to get overwhelmed by all the moving pieces

Exactly my thoughts! Isn't there something (open source and as good as Prometheus+Grafana) that doesn't have as many moving parts as the stack used by OP? I can imagine there are many use cases for that: from side projects (homelabs) to small startups that don't have huge distributed systems, but still need monitoring (without relying on third-parties).

Ideally, my setup would be:

- install an agent in each server I'm interesting in gathering metrics from. In this regard, Prometheus works just fine

- one service to handle logs/metrics/traces ingestion and that allows you to search and visualize your stuff in nice dashboards. Grafana works, but it doesn't support logs and traces out of the box (you need Loki for that)

So, basically 2 pieces of software (if they can be installed by just dropping a binary, even better)

Re:Show HN: Homelab Monitoring Setup with Grafana by gmuslera on Wednesday June 07, 2023 @11:56AM (#36228552)
Telegraf is a single agent that collects a nice amount of metrics and send it to many databases. I prefer to use telegraf (and scripts) to collect the metrics into influxdb and then grafana.

Telegraf have some log parsing/extraction functionality, but for something more generic promtail+loki would be better.

Re:Show HN: Homelab Monitoring Setup with Grafana by alexose on Wednesday June 07, 2023 @11:59AM (#36228607)
My simple-as-dirt solution is generally to use InfluxDB + Grafana. InfluxDB provides a nice HTTP interface that all of my devices simply POST to. I write all the queries myself, because I find that it's a heck of lot easier than to track down individual agents/plugins that actually work.
Re:Show HN: Homelab Monitoring Setup with Grafana by dracyr on Wednesday June 07, 2023 @12:02PM (#36228656)
I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have.

Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup.

A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics.

- [1] https://vector.dev - [2] https://quickwit.io/ - [3] https://log-store.com/

Re:Show HN: Homelab Monitoring Setup with Grafana by esafak on Wednesday June 07, 2023 @02:34PM (#36231334)
Does using vector commit you to DataDog in any way?
Re:Show HN: Homelab Monitoring Setup with Grafana by nullify88 on Wednesday June 07, 2023 @03:14PM (#36231975)
Not at all.
Re:Show HN: Homelab Monitoring Setup with Grafana by francoismassot on Wednesday June 07, 2023 @03:51PM (#36232531)
Nice experiment.

Side note: it should be possible to tweak some config parameters to optimize the memory usage or cpu usage of quickwit. Ask us on the discord server next time :)

Re:Show HN: Homelab Monitoring Setup with Grafana by dracyr on Thursday June 08, 2023 @02:21AM (#36237975)
Thanks!

Yeah, I was a little bit surprised it was so close. And I've been using tantivy (the library which powers quickwit afaik) in another side project where it used comparatively less.

Might jump in there then as an excuse to fiddle a bit more with the homelab soon then :)

Re:Show HN: Homelab Monitoring Setup with Grafana by prabhatsharma on Friday June 09, 2023 @08:20PM (#36266168)
You should try OpenObserve. It combines logging, metrics and dashboards (and traces) in one single binary/container
Re:Show HN: Homelab Monitoring Setup with Grafana by prabhatsharma on Friday June 09, 2023 @08:20PM (#36266171)
Re:Show HN: Homelab Monitoring Setup with Grafana by Cyph0n on Wednesday June 07, 2023 @12:27PM (#36229097)
I use Telegraf (collector) + Influx (storage) + Grafana (visualization and alerting). Telegraf is amazingly simple to use for collection and has a ton of plugins available.
Re:Show HN: Homelab Monitoring Setup with Grafana by EddieEngineers on Wednesday June 07, 2023 @01:00PM (#36229633)
+1 for Telegraf (with Prometheus and Grafana), rolled out a monitoring stack for our internal network in something like 2 days when a colleague had been manually checking `top` for years each morning. Huge benefit.
Re:Show HN: Homelab Monitoring Setup with Grafana by sohooo on Wednesday June 07, 2023 @04:27PM (#36233063)
I also started with that stack, but swapped out InfluxDB for Postgres + TimescaleDB extension, which adds timeseries workflows (transparent partitioning, compression, data retention, continuous aggregates, …).

I found InfluxDB to be lacking in terms of permissions management, flexibility regarding queries (SQL, joins), data retention, ability to debug problems. In Postgres, for example, I can look into the execution plan of a statement, log long running queries, and so on.

Telegraf as an agent is very flexible; it has input plugins for every task I could want, and besides it’s default „pull workflow“ (checks on defined interval) I also like to push new metrics directly to the Telegraf inputs.socket plugin from my scripts (backup stats, …).

Re:Show HN: Homelab Monitoring Setup with Grafana by justin_oaks on Thursday June 08, 2023 @03:34PM (#36247482)
How do you get data from Telegraf into Postgres/TimescaleDB?

I was interested in swapping out InfluxDB, but it turned out to be somewhat difficult to send data from Telegraf to Postgres. It's not as simple as making an HTTP post, like you can do with InfluxDB.

Re:Show HN: Homelab Monitoring Setup with Grafana by nullify88 on Wednesday June 07, 2023 @03:27PM (#36232164)
The closest thing may well be Elasticsearch (and Kibana for visualisation), if you are fine with the Elastic license. As its document format is very flexible, it can be used to store logs, metrics, and traces. It'll be a solution inferior to specialised tools like Prometheus, Mirmir, Tempo, though. And some may be put off by the difficulty of running Elasticsearch.

For example, Elastic has various *beat agents that are able to ship various logs, metrics and traces to Elasticsearch and automatically generate dashboards. An all-in-one agent like Telegraf or Vector can also work, consolidating multiple agents.

Re:Show HN: Homelab Monitoring Setup with Grafana by computershit on Wednesday June 07, 2023 @11:19PM (#36236746)
> Grafana works, but it doesn't support logs and traces out of the box

Grafana doesn't support anything out of the box by that logic. Before you get any viz in Grafana you have to add a data source cmon.