Re:Bring down a poorly deployed WordPress (and how to stop i...
by bradleyland on Saturday February 05, 2011 @01:42PM
(#2183657)
Our application runs an Apache MPM/Passenger/MySQL stack. We conduct real-time purchasing events with anywhere from 5-25 sellers bidding in an event that may have 200+ line items spread out over several lots. Using the "minimum viable product" philosophy, the bidder interface relies on AJAX polling at a 3 second interval to get updated information (yes, we have plans to move to something different [Socket.io, actually]). This means that our app gets hammered during an event. All of this runs within 1 GB of RAM on our VPS, and that includes 6 Passenger instances and MaxClients set to 150 (GASP!). Here's a sample of our passenger-memory-stats output during a large-ish event:
I don't know what the author's experience has been, but I've seen the recommendation to trim Apache back to 10-15 MaxClients all over the place, and when we took that advice, the results were catastrophic to app performance. We almost fell apart during an event because of slow clients tying up Apache processes. We did plenty of benchmarking prior to going live, but getting a picture of real world performance is a lot harder than running `siege -c 250 http://hostname` against your server. If you're going to trim Apache back that far, I'd recommend turning off KeepAlive altogether. You want that Apache process free immediately. If you are forced to run a config this tight, it's likely that you can afford the CPU and interrupt overhead of setting up a new TCP/IP session, but you can't afford the extra processes lying around idle, sucking up memory and MaxClient slots. In other words you're heavily memory bound.
So let me be clear about this: Trimming MaxClients to 10-15 is horrible advice if you don't understand what your actual memory usage (maybe you can run more) is and the impact of slow clients. I trimmed our Apache modules back to a bare minimum, resulting in an average Apache process size of 0.45 MB. Yes, less than 512 KB per Apache process. We run 'MaxClients' 150 with zero fear of running out of memory. If you're running a PHP based site, your Apache process size is going to be larger, but you had better know what that is before you start tweaking your MaxClients config.
I should probably write a blog post about tweaking Apache settings, because I've learned a lot of "hard knocks" lessons, not the least of which is "understand your web server's memory usage", but here are some quick tips in the mean time:
* Unload modules you don't need! Saving a couple megabytes of memory across 100 Apache processes means saving 200 MB of memory.
* To understand how much memory Apache processes are using, DO NOT rely on `top`. Have a look at the way `passenger-memory-stats` calculates real memory usage [1].
* Understand the output of `free -m`, especially the "-/+ buffers/cache" line.
* Do not rely on benchmarking tools like siege to tell you the whole story. Slow clients are difficult to account for.
* If you have to run a limited number of MaxClients, or you run up against your MaxClients limit frequently, try turning off KeepAlive [2].
If you follow those tips, you'll see a dramatic improvement in the accuracy of your Apache config for your environment, because that's what it comes down to. Apache is a fantastic web server, but it has to be configured to fit your environment. I don't have any dislike for nginx, but a lot of the nginx recommendations I see are born out of the inability to properly configure their existing web server. If you don't already know/understand Apache, I wouldn't hesitate to make the jump to nginx. It is also a fantastic web server. Hopefully this has helped someone.
So let me be clear about this: Trimming MaxClients to 10-15 is horrible advice if you don't understand what your actual memory usage (maybe you can run more) is and the impact of slow clients. I trimmed our Apache modules back to a bare minimum, resulting in an average Apache process size of 0.45 MB. Yes, less than 512 KB per Apache process. We run 'MaxClients' 150 with zero fear of running out of memory. If you're running a PHP based site, your Apache process size is going to be larger, but you had better know what that is before you start tweaking your MaxClients config.
I should probably write a blog post about tweaking Apache settings, because I've learned a lot of "hard knocks" lessons, not the least of which is "understand your web server's memory usage", but here are some quick tips in the mean time:
* Unload modules you don't need! Saving a couple megabytes of memory across 100 Apache processes means saving 200 MB of memory.
* To understand how much memory Apache processes are using, DO NOT rely on `top`. Have a look at the way `passenger-memory-stats` calculates real memory usage [1].
* Understand the output of `free -m`, especially the "-/+ buffers/cache" line.
* Do not rely on benchmarking tools like siege to tell you the whole story. Slow clients are difficult to account for.
* If you have to run a limited number of MaxClients, or you run up against your MaxClients limit frequently, try turning off KeepAlive [2].
If you follow those tips, you'll see a dramatic improvement in the accuracy of your Apache config for your environment, because that's what it comes down to. Apache is a fantastic web server, but it has to be configured to fit your environment. I don't have any dislike for nginx, but a lot of the nginx recommendations I see are born out of the inability to properly configure their existing web server. If you don't already know/understand Apache, I wouldn't hesitate to make the jump to nginx. It is also a fantastic web server. Hopefully this has helped someone.
1 - https://github.com/FooBarWidget/passenger/blob/master/bin/pa...
2 - http://serverfault.com/questions/86550/apache-keep-alive-or-...