Re:All my servers have an 8 GB empty file on disk
by geocrasher on Thursday March 25, 2021 @05:54PM
(#26585956)
> I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.
I work in hosting too, and have been for a long time. I feel ya.
Re:All my servers have an 8 GB empty file on disk
by dilyevsky on Thursday March 25, 2021 @05:55PM
(#26585972)
Slack warning/ticket at 75%, page at 85% (to oncall obviously). Don’t let user workload crap into your root partition. I’ve been doing this for over 10 years and managed many thousands of nodes and literally don’t recall full disk problem unless it was in staging somewhere where monitoring was deliberately disabled.
Re:All my servers have an 8 GB empty file on disk
by berkes on Friday March 26, 2021 @09:39AM
(#26591987)
Your requirements for "proper monitoring" are not everyones requirements.
On a current gig, we host at heroku. Our monitoring is all about 95th percentile response-times, secondary services, backlogs, slow-queries and whatnots. For another job, "disk space filling up" is important. Again another job will need to monitor email-delivery-rates and so on and so forth.
You have it mail you when it goes over 80% disk usage (and what if you are on holiday)? Does it mail all colleagues? Who picks it up (I thought Bob picked it up, but Bob thought Anne picked it up. So no one did)? Does it come and wake you in person when it reaches 92%?
Will this catch this async job that fails (but should never) in an endless loop but keeps creating 20MB json files as fast as the disk allows it to?
Is it an alerting that finds anomalies in trends? Will it be fast enough for you to come online before that job has filled the disk?
I've been doing a lot of hosting management and such. And there is one constant: all unforeseen issues are unforeseen.