News for hackers. Stuff that matters.
Re:Busiest hour ever in the history of Gov.uk 17M page reque... by londons_explore on Tuesday January 05, 2021 @01:21PM (#25648861)
I have worked at a company where every 500 error is a reason to go check the logs and try to reproduce... We would literally page an engineer.

I've also worked at a company where "as long as the error rate stays under 0.1%, it's fine". As long as each app server doesn't have more than 1 crash per week it's fine.

I can tell you that engineers at the first company end up finding all kinds of super rare bugs, usually in the OS, platform, malloc library, load balancer, etc. They then commit fixes for those bugs which ends up helping the latter company...

Re:Busiest hour ever in the history of Gov.uk 17M page reque... by treve on Tuesday January 05, 2021 @01:59PM (#25649435)
It's an interesting take. We definitely don't have the capacity & skills to be in the first company, although it would be awesome.

I think most places I worked we had no choice but to be in the second company, although I think the rate is a bit below .1%. I'm grateful for all the people hunting down these bugs and making the world better for the rest of us.

I remember around 2006, we still found PHP segfaults that were critical enough I reported at least a dozen. Everything feels so much more stable today (at least on that layer).

Re:Busiest hour ever in the history of Gov.uk 17M page reque... by khaless on Tuesday January 05, 2021 @02:57PM (#25650198)
While it's nice to get paged, and look at every 5xx error; it doesn't really scale all that well once you get past a certain point, particularly if your application is gracefully degrading.

That said, I love the wisdom in your comment that you find all sorts of super rare bugs, or conditions that could seriously effect performance, or availability if they become more common (which they often do). Past a point, I've found that an approach which works well is to encourage engineers/operators to drive by metrics, and pay close attention over time to p100's (max), as you've suggested with your 500 errors. Lots of goodies can be hidden behind them, just like you've found with the 500 errors.

Re:Busiest hour ever in the history of Gov.uk 17M page reque... by shinji97 on Tuesday January 05, 2021 @02:51PM (#25650114)
curious if you can share a little more on the first company you mentioned, what was their area of business?