News for hackers. Stuff that matters.
Re:Riemann – A network monitoring system by hendry on Tuesday January 19, 2016 @08:47AM (#10930664)
Anyone use http://prometheus.io/ ? wdyt
Re:Riemann – A network monitoring system by bovermyer on Tuesday January 19, 2016 @12:07PM (#10932042)
We switched from using Riemann to Prometheus, and we're very happy with the decision. Prometheus has been much easier to work with.
Re:Riemann – A network monitoring system by MayMuncher on Tuesday January 19, 2016 @09:27AM (#10930911)
I don't like it since the master basically does a curl on each server to get the info, so it doesn't work behind a firewall without tons of issues
Re:Riemann – A network monitoring system by zimpenfish on Tuesday January 19, 2016 @09:35AM (#10930958)
That's the reason I haven't bothered with Prometheus - pull systems make no sense to me since you have to configure a single place with perfect* knowledge of your system rather than just pushing local knowledge to a collector.

(Ok, you do need some knowledge at the parent if you want to raise alerts but you'd need that anyway.)

Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @10:01AM (#10931151)
You need a place with all that information anyway, otherwise how do you alert on something being missing?

If you can't easily do that with your existing infrastructure, you should fix that first. I've written about this at http://www.robustperception.io/you-look-good-have-you-lost-m...

Re:Riemann – A network monitoring system by zimpenfish on Tuesday January 19, 2016 @10:13AM (#10931249)
If a stream of data stops appearing, then you can alert. You don't need to pre-configure the existence of that stream (although if you might never get data from an object, this obviously is a failure you won't catch.)

Your article is a good one but in my experience, many companies are still many years from being able to implement that kind of database:machine knowledge consistency.

Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @10:47AM (#10931453)
You can only alert in that case if the data starts, and there's no transient issues preventing your monitoring working around the time it stops.

Alerting based on state changes is fragile, it's better to compare against what you expect to see.

Re:Riemann – A network monitoring system by vruiz on Tuesday January 19, 2016 @11:34AM (#10931769)
You can leverage service discovery to make your configuration dynamic: http://prometheus.io/blog/2015/06/01/advanced-service-discov...
Re:Riemann – A network monitoring system by robzienert on Tuesday January 19, 2016 @09:37AM (#10930977)
That's what the Prometheus push gateway is for. While prom itself is poll-based, you can extract data from servers behind a firewall by pushing metrics from an agent into the gateway. No crazy issues.
Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @10:01AM (#10931157)
The general advice is to run Prometheus behind the firewall, you want as few things between your monitoring and what you're monitoring as possible. This makes it more resilient.
Re:Riemann – A network monitoring system by lobster_johnson on Tuesday January 19, 2016 @06:43PM (#10935004)
It supports push, though.
Re:Riemann – A network monitoring system by lobster_johnson on Tuesday January 19, 2016 @06:42PM (#10934999)
We're using Prometheus. Very happy, though the collection side of things aren't as smooth as we'd like it.

The lack of a plugin system, and the reliance on HTTP to collect, means writing small collectors is a pain. You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ.

We opted for the "text directory" way, where we populate a directory with .prom files that node_exporter automatically picks up. It's not ideal: It means a whole bunch of collectors run via Cron jobs, which themselves need to be monitored; it means if we remove a collector, we also have to clean up its .prom files; and in the end it meant we had to invent our own little plugin system in order to avoid writing a lot of boilerplate code needed to manage the different collectors we use. We'd love to share our collectors, but due to that last point, our collectors are less reusable than we'd like.

Prometheus itself has been quite stable, but it still has some rough edges:

* If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database (e.g., see [1]).

* There's no way to do snapshots of the database.

* The team is rather cavalier about backwards compatibility. We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production. The alert manager was also written from scratch recently, with a whole new config format. With several releases, every tool has had its command line flags changed ever so subtly, too.

* The lack of packages (Debian/Ubuntu in our case) is also problematic. Fortunately I've got a script now that grabs a release and bundles a .deb from it, but I'd vastly preferred real releases.

* No syslog support is not acceptable in this day and age. Our Upstart scripts spawns a "logger" subprocess to catch stderr. Not everyone is running under Docker.

[1] https://github.com/prometheus/prometheus/issues/877

Re:Riemann – A network monitoring system by bbrazil on Tuesday January 19, 2016 @08:12PM (#10935367)
> The lack of a plugin system

We have many ways to plugin to Prometheus across the ecosystem, the textfile collector you're using is one of them.

> You can't run 23 different daemons, each on their own port, to collect stats from things like PostgreSQL stats or RabbitMQ.

There's no fundamental challenge with this approach. If you've got good basic infrastructure, particularly configuration management, the rollout of each should be a small operational task. If it's a major challenge, then your problem probably isn't with the Prometheus architecture.

> it means if we remove a collector, we also have to clean up its .prom files

There's several problems arising from this approach, this is one of them. You can also expect odd artifacts in graphs.

The textfile collector is only intended for machine-level metrics, by putting service level metrics in there you're missing out on a big win of Prometheus by thinking in terms of machines rather than services.

Fighting against the architecture means you're not getting the maximum benefits from Prometheus, this would be easier with exporters and service discovery.

> which themselves need to be monitored

Are you aware that the node exporter exports the mtime of all the textfile collector files? That's there to make monitoring of them easier.

> If anything goes wrong with its database files, it tends to just crash, and the only way out is to wipe the entire database

As far as we're aware, the only way that happens is if you run out of disk space. If you've evidence otherwise please let us know, so we can prioritize accordingly.

> We've experienced at least one version upgrade where they changed the database format and didn't provide any upgrade tools, so people were forced to start their metrics history from scratch. I know that it's pre-1.0, but still, they knew perfectly well that people were running it in production.

We broke backwards compatibility in the storage format once, and there's no plans to do so again. The core developers who were all running it in production didn't see it as worthwhile to write a converter, and noone else stepped up.

> The alert manager was also written from scratch recently, with a whole new config format.

The old alertmanager has always been flagged as very experimental, as it was a functioning PoC. The rewrite was always been on the cards, and this came up regularly.

This is all part of evolving the system to be better for everyone. If we tried to keep perfect backwards compatibility then we couldn't remove warts, bugs and misfeatures. We aren't afraid to deprecate where it makes sense to do so, and have transition plans where practical.

> The lack of packages (Debian/Ubuntu in our case) is also problematic.

There are packages in Debian proper, and nightlies at http://deb.robustperception.io/

> No syslog support is not acceptable in this day and age.

That's in the latest versions.

The high level problem is that there's so many different ways to do logging that we can't sanely support them all. For every X there is someone who thinks it's essential.

Re:Riemann – A network monitoring system by lobster_johnson on Tuesday January 19, 2016 @10:47PM (#10935939)
> If you've got good basic infrastructure [...]

We do have good basic infrastructure, thanks. We use Puppet and have a decent deploy system that performs atomic deploys from Git.

We also do think in terms of services. But the exporter has to run somewhere. About half of our exporters are machine-specific (reads local stats from files or proc or whatever), about half run on the Prometheus node itself and talk to services like ElasticSearch or Postgres.

The problem is operational overhead of maintaining a dozen daemons per box, each with its own allocated port. It's not rocket science, just annoying. What could be a small script becomes something unnecessarily big. That's why we're sticking to "textfile" for now. When I have time, my plan is to write a small HTTP server that spawns plugin as subprocesses that emit their metrics via stdout, which seems like a much more reasonable, low-maintenance solution, and something node_exporter ought to support in the first place.

Re:Riemann – A network monitoring system by bbrazil on Wednesday January 20, 2016 @03:52AM (#10936778)
> (reads local stats from files or proc or whatever)

If there's useful stats from /proc we're missing, we accept PRs.

> about half run on the Prometheus node itself and talk to services like ElasticSearch or Postgres.

That doesn't sound right, those two should run on the ES/Postgres nodes.

Only things like the blackbox exporter and snmp exporter should be considered for running beside Prometheus.

> When I have time, my plan is to write a small HTTP server that spawns plugin as subprocesses that emit their metrics via stdout, which seems like a much more reasonable, low-maintenance solution, and something node_exporter ought to support in the first place.

We went with the textfile collector approach, rather than reinventing cron.

Re:Riemann – A network monitoring system by lobster_johnson on Wednesday January 20, 2016 @08:17PM (#10942729)
Sorry, I misspoke. Only non-machine-specific collectors run on the Prometheus machine.

> rather than reinventing cron.

But Prometheus already has a job interval setting. With text files emitted with cron jobs, you get the silly effect where metrics are produced at intervals which don't correspond with the job interval.

Re:Riemann – A network monitoring system by uberneo on Tuesday January 19, 2016 @12:23PM (#10932154)