Re:GitLab Database Incident – Live Report
by devdas on Wednesday February 01, 2017 @06:58AM
(#13540010)
Monitor the rate at which the disk is filling up, and extrapolate that to when it will hit 90%. If that time is outside business hours, alert early. If current time is not in business hours, alert later if possible.
[ Parent
| on HN ]
Re:GitLab Database Incident – Live Report
by bpchaps on Wednesday February 01, 2017 @12:25PM
(#13542713)
How does this help in situations where something rogue starts filling the disk? The idea makes sense in theory, but in practice, it doesn't work out that well. Ops work is significantly harder than many devs think..
Re:GitLab Database Incident – Live Report
by kelnos on Wednesday February 15, 2017 @01:50AM
(#13650227)
> Ops work is significantly harder than many devs think
No, it's not (I've done both). Ops is about process, and risk analysis and mitigation. Yes, there's always the possibility that something can go rogue and start filling your disk. That shouldn't be remotely common, though, if you've built your systems properly.