The problem in this case was twofold:
- admin had the job to implement database backups. He didn't factor in backup size when allocating disk space. So this was wntirely his own fault.
- the database does store certain transactions for a certain period, so this grew initially until it it setteled at a certain level. Because the margin of storage was slim, this caused the problem
Because most of our cases where that happens was either lack of planning or lack of communicating that plan. By far most common one was "neither dev nor client knows the data volume in longer period". Which is fine as long as that's also communicated, but that's also often a problem.
But I'm not denying of course that there are just shitty incompetent ops departments, just for the other customer we had dealing with ops department that had:
* backup storage (which was some remote FTPS server IIRC) provisioned so slow the backup wouldn't copy within 24 hours. And the backup size was below TB. * weeks long delays with any resize.
But, unlike Toyota, we do have disk space alerts.
Sometimes the problem is also entirely political, the management needs to tell client and charge them for more storage and won't accept the change till that happens. Meanwhile clock is ticking...