News for hackers. Stuff that matters.
Re:A ChatGPT mistake cost us $10k by asveikau on Monday June 10, 2024 @12:27AM (#40630144)
I think deploying and then going to sleep is the red flag here. They should have deployed the change at 9am or something and had the workday to monitor issues.
Re:A ChatGPT mistake cost us $10k by bruce511 on Monday June 10, 2024 @12:41AM (#40630224)
While I agree, it wouldn't have helped. The first n sign-ups per commit worked. So the problem didn't manifest during the day, only when they stopped committing.

Now granted, if they'd carried on doing stuff (but not redeployed production) it may have shown up in say mid-aftenoon, or not depending on volume.

And of course the general guideline of not deploying anything to production early in the day, or on Friday, is still valid.

Re:A ChatGPT mistake cost us $10k by lolinder on Monday June 10, 2024 @12:59AM (#40630312)
You can deploy and go to sleep if you have monitoring and alerting and someone getting paged. It shouldn't be a human monitoring for issues anyway, so the only reason to choose 9am over bedtime should be that you don't want to risk a late night page, not that someone will actually be checking up actively on the deployment.
Re:A ChatGPT mistake cost us $10k by asveikau on Monday June 10, 2024 @01:08AM (#40630353)
For a big launch, you can have your entire team looking at issues in realtime, rather than a small number of people responding to pages in the middle of the night. I think if it's worth it to you there's a big difference.
Re:A ChatGPT mistake cost us $10k by owl57 on Tuesday June 11, 2024 @02:16AM (#40643060)
That is, if you know what to monitor. When you launch an entirely new feature, you often don't know all the ways it will break.

And it's not trivial to set up a correct alert for this one. Simple HTTP 5xx threshold wouldn't work because it's high enough to not wake you up whenever cloud provider restarts Postgres, it's too high to catch this. You need either per-endpoint failure rate alerts or something else more clever.