The subcritical alerts I think of are more things like "Well, the database is _getting_ full, but it's not full yet." Or to borrow someone else's example, "we put in this daemon restarter when it was dying once a week; now it's dying every few minutes and we're only surviving because our proxy is masking the problem but soon it's going to take the whole site down."
The other useful tip I have is to put URLs to internal wikis and/or tickets in the alert body. We write documentation for these to a 3AM standard: if I can't understand it immediately after being woken up at 3AM, it's not clear or actionable enough.
There is no reason anyone should ever run out of disk space if they alert on the trending rate of disk space [rather than the actual amount of disk space used]. But this applies to so, SO many things other than simple resource exhaustion. Seeing the trends is useful to alerts, but it's also useful to humans who can review them weekly and plan for the future.
If you (or anyone else reading) ever want to talk about what "this will be a problem soon" might look like in the future drop me a line: [email protected]
This is an excellent point that is missed in most monitoring setups I've seen. A classic example is some request that kills your service process. You get paged for that so you wrap the service in a supervisor like daemon. The immediate issue is fixed and, typically, any future causes of the service process dying are hidden unless someone happens to be looking at the logs one day.
I would love to see smart ways to surface "this will be a problem soon" on alerting systems.