Why in the world would you want to manage (which I'm reading as throttle) the ingest rate of a monitoring service? That strikes me as a recipe for missing important events.
Monitoring should be a fairly stable load rate. If you're setting up monitoring, and your machine can't handle the load of all the data points coming in, you need to shard out, not drop data points.
> Active polling scales _better_ than the alternative
Exceptional claims require exceptional evidence. Personally, I have never heard of anything that is actively pinging outside services performing better than receiving and processing data passively.
Prometheus will scale better than Nagios running active checks since it won't be using subprocesses, but it is still going to require more overhead than a service receiving passive reports.
A single big Prometheus server can easily do millions of time series, and in an older record, 800,000 samples stored per second. You could monitor e.g. 10,000 hosts with that with quite some detail, and the bottleneck is still not the pull aspect.
We also have an FAQ in Prometheus about why we prefer pull: https://prometheus.io/docs/introduction/faq/#why-do-you-pull...?
In my experience, pulling is operationally much nicer than pushing, and I've worked with both. It also gives you somewhat less of an accidental DDoS exposure.
> [Grafana+Ichinga] they're so rock solid (and scale like mad) I have a hard time not recommending them.
As a Prometheus developer I have seen a significant number of users who moved from Graphite because they found it doesn't scale and was far from rock solid for them, requiring regular manual care and feeding. By contrast Prometheus seems to be working pretty well for them at what we would consider to be a moderate load. I've heard similar about Nagios/Ichinga.
Push vs. pull is largely not relevant for scaling (pull is slightly better in this regard, but only slightly). I've been involved with some extremely large scale monitoring systems, and the fact they were pull was never relevant to scaling them.
May I ask what you consider to be a high level of scale?
> It won't scale without a number of workarounds.
There are very very few systems who won't scale without workarounds. That's the nature of scaling a non-trivial system.
If you need to shard, no other binaries are involved, just a couple of changed configuration files and some SSL certs. Passive reports can go to any node; a simple load balancer will work fine without any form of hashing or federation required.
> There is nothing inherent in a polling model that limits its ability to scale
Aside from the additional processing requirements, such as SSH, NRPE, subprocesses, etc, you are also limited in that a polling process must be told in advance of new systems or services, whereas it's fairly easy to just have a new service or system start reporting and be immediately monitored.
How do you re-combine and alert on data for one service that got reported to multiple nodes? This would imply to me a database of some form, which is going to need hashing or federation or other distributed systems approaches to be scaled out.
> Aside from the additional processing requirements
Push or pull, there's additional processing requirements when you scale. It's the same bytes on the wire and broadly the same amount of processing power required.
> you are also limited in that a polling process must be told in advance of new systems or services
That's not a scaling limit, that's a more fundamental issue that isn't different between push and pull.
For a push system you need to have a list of all systems and services in order to be able to alert on systems that never reported, or are no longer reporting.
Pull works fine at 2 million machines" is a great statement to make, but it would be much stronger with more details. How many machines are doing the pulling? How often? Are they using subprocesses or threads or greenthreads? How do they handle timeouts? How many metrics per machine? How many pulls per metric grab?
My preferred method is Icinga2 (a Nagios clone with better configuration and clustering built-in) with reports coming in via passive NSCA. Toss in Graphite (or I'm warming up to Grafana on Influx) with some ability to write alerts against those reported metrics, and you're as close to ideal as I can come up with.
Of course, that requires a fair bit of up-front knowledge to stand up and operate, but they're so rock solid (and scale like mad) I have a hard time not recommending them.