News for hackers. Stuff that matters.

A comment on Building resilient services at Prime Video with chaos engineering. Parent comment

Re:Building resilient services at Prime Video with chaos eng... by ses1984 on Monday August 24, 2020 @09:31AM (#24260451)
>"Have you tested how the system behaves when the underlying instances have a sustained CPU spike?"

You didn't really address this question, you addressed a different question, which is a traffic spike.

>Also, I'd say my customers can endure it if page load time goes up from 100ms to 200ms. So in my opinion, there is little need for auto-scaling for most companies.

100ms to 200ms average? What about the tail? Your app might go from P99 - 500ms to P95 - timeout. That's when you'll lose customers.

[ Parent | on HN ]
Re:Building resilient services at Prime Video with chaos eng... by fxtentacle on Monday August 24, 2020 @03:06PM (#24264207)
If the underlying hardware is a bare metal server, it won't magically turn slow and have a CPU spike. That problem is caused noisy neighbor and kind of exclusive to clouds.

Well, with the 2x example, my app might get from a 1s P99 to a 2s P99 which feels slow, but is still doable. Again, those timeouts are usually introduced by cloud infrastructure. For example, if you use nginx outside of Heroku, it won't have a 30s timeout for file downloads.

[ Parent | on HN ]
Re:Building resilient services at Prime Video with chaos eng... by ses1984 on Monday August 24, 2020 @03:12PM (#24264267)
Your own instances can have an unexpected CPU spike.

Even if you're running on bare metal I find it hard to believe you don't have a layer with short timeouts between your front and backend.

[ Parent | on HN ]
Re:Building resilient services at Prime Video with chaos eng... by fxtentacle on Monday August 24, 2020 @04:16PM (#24265025)
Why would I? I have redundant 1GBit LAN cables between front end, back end, and database servers.
[ Parent | on HN ]
Re:Building resilient services at Prime Video with chaos eng... by ses1984 on Monday August 24, 2020 @06:17PM (#24266165)
Because it's bad ux for your users to see a spinning loading icon forever.
[ Parent | on HN ]
Re:Building resilient services at Prime Video with chaos eng... by fxtentacle on Monday August 24, 2020 @06:36PM (#24266324)
And a timeout error would be better?
[ Parent | on HN ]
Re:Building resilient services at Prime Video with chaos eng... by ses1984 on Monday August 24, 2020 @09:55PM (#24267491)
In my experience, yes, a lot better.
[ Parent | on HN ]