Re:Netdata: Open-source real-time monitoring platform
by jeffbee on Wednesday April 21, 2021 @12:44PM
(#26892021)
Based on my experience a dashboard that you only use when you know where and when the problem occurred is incredibly useful, and the lack of one can be very frustrating. While you of course need a systematic approach to incident detection, you also need comprehensive eyes-on-glass dashboards during your investigations. "Anomaly detection" is much spoken of but generalized anomaly detection doesn't exist. You still need skilled operators to just have a look around in many cases.
An example, drawn from several major incidents in my career. You get an alert, you narrow it down to a process or machine, you evict the machine from your serving population to remediate the incident, but how do you keep it from recurring? The anomalous thing isn't apparent in your monitoring data, so it must be among the bazillion statistics that a running system exposes, but which you can't afford to collect and monitor on a per-host, per-container, per-process level of detail. That's when you want something exactly like netdata!
An example, drawn from several major incidents in my career. You get an alert, you narrow it down to a process or machine, you evict the machine from your serving population to remediate the incident, but how do you keep it from recurring? The anomalous thing isn't apparent in your monitoring data, so it must be among the bazillion statistics that a running system exposes, but which you can't afford to collect and monitor on a per-host, per-container, per-process level of detail. That's when you want something exactly like netdata!