Background: we moved all our pan-US servers ( Windows, Unix, AS/400 and MF ) to UTC back in the early 2000s, instead of following the local TZs.
I didn't experience any particular instance when I wanted to convert a server incident to wall-time. Since all events were scheduled in UTC it was a direct look-up between (1) unusual things happening on server and (2) any jobs scheduled at that time. What time it was in the office or in the customer's TZ didn't really matter.
The only time we converted to wall-time was internally within teams to determine when our maintenance shifts would start. With globally-outsourced support teams, the server load charts dictated the UTC time to start the work which was communicated to the appropriate teams who performed local conversion for their shifts. Once they were in the office, everything was executed on UTC.
Not this one:
>> I have (in the past), seen servers where the system timezone is set to PST, but the database is set to UTC, causing more crazy bugs, because for 8 hours a day, calling now() in the database has a different date from the application that talks to it.
If your servers and applications are both on Arizona time, it doesn't manifest itself.
If you're exposing an API to (or otherwise sharing data with) third parties, they're unlikely to have standardized on your localish timezone, unless they happen to share it. If you're consuming a third party API (or otherwise consuming data from them), they're unlikely to have standardized on your localish timezone, unless they happen to share it.
If you live in a disconnected bubble, your localish timezone might be consistent enough. You probably don't live in a disconnected bubble. Your localish timezone is not consistent. It is not "Universal". It is not UTC.
$ TZ='<UTC-7>+7' date -R
Sat, 16 Jul 2016 11:04:52 -0700
Or use Sonora in Mexico: $ TZ=:America/Hermosillo date -R
Sat, 16 Jul 2016 11:05:39 -0700
Another option is Pitcairn Island, which is -0800 all year. $ TZ=:Pacific/Pitcairn date -R
Sat, 16 Jul 2016 10:06:23 -0800
though the standard abbreviation, PST, is probably confusing.
If your company is entirely located in California or some other Pacific Time location, consider putting your servers on Arizona time. There's no daylight saving, and for most of the year, the server time will match your wristwatch. In the winter, when California is on PST, the servers will only be off by an hour.
I find that much more tolerable than the hassles of UTC servers, where you can't instantly translate between server time and wall time in your head, and server midnight rolls over in the middle of the California day, and thus datestamped logfiles don't match up at all with your wall calendar.
Edit: If you're going to downvote, please post a comment explaining why, so that readers can learn something.