News for hackers. Stuff that matters.
Re:A self-killing web site requested by a customer (2011) by pjc50 on Thursday May 07, 2020 @07:46AM (#23101907)
There's an Apache setting to limit that, or at least there was when I last had to deal with this problem 20 years ago on webservers that had less processing power than today's midrange smartphones.

The problem is you have to manually tune the number of max workers a bit based on estimated resource usage.

Re:A self-killing web site requested by a customer (2011) by michaelt on Thursday May 07, 2020 @08:44AM (#23102305)
And even if you've got a really good idea of how many connections you can handle at a time, when problems start happening the patterns might change.

For example, maybe in the average second you see 1 login request (CPU-bound password hashing), 10 page views (database connection bound), 100 image views (IO/client speed bound), and 0.01 login sessions being bounced between servers (???? bound). And your server is limited to 111 connections per second.

But when the site goes down? Users can't load images until they've seen pages, and can't see pages until they've logged in, so now you're processing 111 login attempts per second when your tuning assumed a fraction of that.

Re:A self-killing web site requested by a customer (2011) by lathiat on Thursday May 07, 2020 @08:38AM (#23102255)
Part of the problem is that you usually need a lot more Apache workers to process static file requests. But limit the number of expensive PHP invocations

This was pretty much not easily possible (with suphp, mod_php) until FPM which has a per site worker pool.

Solved a lot of problems for us as a shared hosting provider.

Re:A self-killing web site requested by a customer (2011) by ColinWright on Thursday May 07, 2020 @11:17AM (#23103949)
When did that happen? I'm interested in the timeline.
Re:A self-killing web site requested by a customer (2011) by atsaloli on Thursday May 07, 2020 @12:30PM (#23104849)
I dealt with a similar issue in 2001. About 6 mid-range Sun servers (like small refrigerators in size), running Netscape Application Server, with the load distributed using round-robin DNS (A record round-robin). Due to some DNS servers caching the A record longer than intended, it would create uneven distribution of load -- too much on one server and it'd go bad, requests would start taking too long and it'd start throwing 500 errors or just wouldn't respond. Once it stopped accepting connections, other servers would take on its load (the client would retry to the next IP in the round-robin) but then similar thing would happen to the fifth server, and the rest would follow shortly thereafter in a death spiral.

Restarting was a bear. We had to "warm up" the application server by hitting some of the web pages so caches would get populated etc., before opening the system to public traffic (which, as I recall, we controlled with IP address aliases) or else it would die again.

Finally we put the system behind an F5 load balancer which resulted in a much more even distribution of traffic -- that, coupled with the "warm up" crawl, let to a greatly stabilized system and highest ever (and growing) page views.