News for hackers. Stuff that matters.
Re:12,000 Requests per second with Nginx, PHP and Memcached by rkalla on Tuesday February 15, 2011 @12:28AM (#2220273)
For 4 years I tried to run my WordPress site off of Apache + PHP.

For the first 3 years it was Apache+Prefork+mod_php+WordPress (default setup for any on RedHat-based or Debian-based setups for the longest time).

Any time I would get a story on Slashdot or Digg the site would die for at least half a day... god I hate it.

I went from a 1GB RAM VPS to a 12GB dedicated machine in 3.5 years trying to get it to stop crashing whenever I would get a flood of traffic and was never able to. I pushed back on the idea of being a Linux sys admin for so long because I didn't want that hassle, but alas, I had to bother with it.

Finally at about the year 4 mark I decided no REAL site on the planet was running with this configuration since it didn't seem to matter the hardware you threw at it (yes I tweaked the Apache setup/mod-list tirelessly to scale with the improved hardware). I finally started digging into how real human beings setup Apache and came across the argument for using the "MPM worker" as opposed to the default pre-fork worker.

Made sense to me; less 30+ MB processes running around answering questions.

After that change, it helped a little... I have no hard numbers on hand, but it felt like a small improvement.

I kept digging and soon ran across the one-hacky-but-now-officially-supported method of using Apache + a family of PHP VM threads pre-launched and called via FastCGI to execute the .php pages from my WordPress site, the computer-science part of my brain loved this idea... the Java-trained side of me suddenly made me realize that prior to this with Prefork and mod_php, every time someone was connecting I was spinning up a new Apache thread and a new PHP VM every single time (please correct this if wrong... this is how I understood it).

With FastCGI I could have a family of say 20-some PHP VM threads living in harmony and responding to Apache constantly asking them questions.

After rolling that change out at about year 4, I noticed a big improvement; maybe about 50%.

At my next Slashdotting the server got REALLY slow, but hung in there; no crashes. I thought it was odd that all that hardware still couldn't host things snappy... it seemed like every other day I was clicking a link off of Hacker News or Reddit front page to some dude's personal blog that was responding very quickly to me and I was positive these people weren't spending $300/mo like I was on dedicated hardware to run their blog.

So I kept digging.

As you guys probably know, when you start searching for what sucks about Apache two things come up more than any other: "use nginx" or "use lighttpd" -- I had read that early versions of lighttpd had some memory leak issues (I think long-since fixed) and had a handful of Ruby friends that loved nginx... so I decided to stay up all night one night and port the site over.

25mins later I was done.

Yea so that was a lot easier than I expected. The only painful part was using some heavy handed redirect logic to convert my WP-SuperCache rules over to nginx (the author wasn't supporting nginx yet, but I think he does not).

I would point out that the server load with nginx running with NO CACHING (WP-SuperCache disabled, all queries execute PHP and perform a MySQL query) was something like 1/4 what my Apache/MPM/FastCGI/PHP/WP-SuperCache-enabled setup was using.

Once I got WP-SuperCache up and running on nginx, the different was stupid-big. The nginx/WP-SuperCache setup was using 1/8th or 1/10th the system resources that the Apache setup had been using.

Soon there after I got Slashdotted with a peak of roughly 400 users on at a time and the server load barely crawled beyond a 1 (screenshot): http://www.thebuzzmedia.com/wp-content/uploads/2010/12/slash...

Overall I couldn't be happier with nginx. I think there are probably people that live in oxygen-rich test chambers inside of military bunkers who were bread to tweak Apache that can optimize it to have comparable performance, but that wasn't me. Out of the box nginx has been fantastic thus far.

And that is my little story related to this subject... for what it's worth.

Re:12,000 Requests per second with Nginx, PHP and Memcached by midnightmonster on Tuesday February 15, 2011 @10:52AM (#2221607)

  > with Prefork and mod_php, every time someone was 
  > connecting I was spinning up a new Apache thread and a 
  > new PHP VM every single time (please correct this if 
  > wrong... this is how I understood it).
Since no one has yet... no, this isn't quite right. Apache in prefork mode maintains a pool of workers, each of which does contain essentially the full webserver and PHP system. These listen for requests and respond to 1 request per process per time. Once a process finishes responding to a request, it starts listening for a new one again. Apache tries to keep at least a few idle processes on hand all the time, so clients don't have to wait, and it will fork new processes to handle more simultaneous connections up to a configured limit. If you get a sudden spike in traffic which then dies down, Apache will start killing excess processes. Processes live until killed as unnecessary, or after serving N requests as configured.

That all sounds pretty reasonable, until you realize that every request, even for styleseets, scripts, images, favicon.ico or robots.txt, is served by one of those fat apache+php processes, which can easily reach 10MB of memory usage each (or a lot more sometimes, depending on your modules). And if Keep-Alive is enabled, or a client is a bit slow, or some part of the Internet gets dodgy, a process can get tied up for a long time, requiring Apache to start more, increasing your memory usage and leading to thrashing and/or hitting the max child processes limit.

Default max is 256 child processes, so easily enough to cause disk thrashing on a 1GB machine, and not enough to fully utilize a much larger machine.

Re:12,000 Requests per second with Nginx, PHP and Memcached by antihero on Tuesday February 15, 2011 @10:11PM (#2224767)
In what way does nginx do this differently? It has workers - are they just lighter and more efficient?
Re:12,000 Requests per second with Nginx, PHP and Memcached by midnightmonster on Wednesday February 16, 2011 @06:33AM (#2225995)
Nginx can have workers, but the default number is one. nginx doesn't need a separate worker per time to handle each request. Anything that happens on a given connection is an event (connected, received some data, finished receiving data, got a response from the backend, closed the connection), and nginx just responds to events in the order received across all connections. It doesn't have to hang out and wait for the client to finish sending it's request, doesn't have to wait for PHP to process the request, just goes about handling the next event until those time-consuming things finish.

If you've looked at using PHP with nginx, you might be thinking of the fastcgi workers. Rather than including PHP in the web server process, fastcgi keeps a pool of PHP processes around, and whenever you get a PHP request, nginx passes it off to the fastcgi backend (then goes about other business until the backend responds). I don't know of any reason that PHP run this way should be faster than as an apache module, but the whole system ends up faster because the web server part is using a lot less memory and CPU. And depending on your site, there are some really low hanging optimizations you can do with nginx. E.g., just need a couple of configuration lines to cache all PHP GET requests for 5 min (unless you have cookie X, if you want). Since nginx seriously will not bat an eye over serving a redditing worth of static files, that means even your unoptimized word press site can stand up to the front page no problem. (Not verified from actually attempting it. But I did setup a simple PHP site on a 512 Linode that served 250k page views per day without excessive disk activity or the CPU usage crossing 10%.)

Re:12,000 Requests per second with Nginx, PHP and Memcached by antihero on Wednesday February 16, 2011 @09:30AM (#2226445)
Aye, I have a 512 Linode, I can get about 1k req/s with nginx/fastcgi and about triple that with caching using memcached :D

That said, I'm trying with httperf and getting much higher numbers (around 38k req/sec :))

Re:12,000 Requests per second with Nginx, PHP and Memcached by mscarborough on Tuesday February 15, 2011 @04:54AM (#2220734)
nginx is great. so is lighttpd.

but so is apache + php + apc + memcached. easy to set up, and with minimal work to write a wrapper, you have a very effective web + caching. don't understand why PHP + APC + memcached gets such short shrift.

Re:12,000 Requests per second with Nginx, PHP and Memcached by mise on Tuesday February 15, 2011 @06:02AM (#2220827)
I got someone to install nginx for me on a not-so-busy server. Twice it filled up /tmp and ran out of inodes, crashing the server. I'm in for peak traffic next month and am afraid it will all come crashing down again :) Oh, and any help you can throw at me, it's appreciated.
Re:12,000 Requests per second with Nginx, PHP and Memcached by ichilton on Wednesday February 16, 2011 @04:13AM (#2225766)
Like others already said, that'll be php creating session files, not Nginx.

Sure it ran out of inodes and it isn't that /tmp is a ramdisk/tmpfs and only small? - you can change the session path in your php.ini by changing/setting session.save_path.

If inodes are the problem, you could also store php sessions in memcached - a google search will give you loads of instructions on this, eg: http://www.dotdeb.org/2008/08/25/storing-your-php-sessions-u...

Re:12,000 Requests per second with Nginx, PHP and Memcached by zepolen on Tuesday February 15, 2011 @08:41AM (#2221109)
Filled up /tmp with what?
Re:12,000 Requests per second with Nginx, PHP and Memcached by bxr on Tuesday February 15, 2011 @09:01AM (#2221175)
I'm going to guess PHP sessions. Off the top of my head its the only /tmp file creating thing I can think of in common web stacks.

If that is the problem mise is facing, you can swap out the storage that PHP uses for sessions (to use something other than files), I'd point out the relevant docs but I have a meeting to get to.

Re:12,000 Requests per second with Nginx, PHP and Memcached by mise on Tuesday February 15, 2011 @11:35AM (#2221870)
I'll look into this. My host Servint says that my inode allocation is "generous", so I wonder if I'm the only person running into this problem with PHP+nginx.
Re:12,000 Requests per second with Nginx, PHP and Memcached by zepolen on Wednesday February 16, 2011 @10:40AM (#2226759)
It has nothing to do with nginx.
Re:12,000 Requests per second with Nginx, PHP and Memcached by mise on Thursday February 24, 2011 @05:32AM (#2257487)
Argh. But it happened precisely twice, only after nginx was installed, and without any traffic peak. :/
Re:12,000 Requests per second with Nginx, PHP and Memcached by sunkencity on Tuesday February 15, 2011 @03:39AM (#2220608)
I'm pretty happy with nginx but I do mostly rails with that. For PHP, the general consensus seems to be fastcgi + lighttpd for high performance environments, but if you're just serving the same page, why not use apache + mod_cache. What improvements over mod_cache would wp-supercache get, you're just serving static files, so why not just add varnish or mod_cache for apache.

That said, serving small files from lighttpd is extremely fast. I set up one high-performance server for small files that served from ramdisk on a linux machine. At 15000 r/s with ab or siege I didn't max out the small instance on amazon. Sceptical about correctly measuring performance at that level though.

Re:12,000 Requests per second with Nginx, PHP and Memcached by jacques_chester on Tuesday February 15, 2011 @04:10AM (#2220648)
Lighttpd had a persistent bug where overloading FastCGI instance of PHP would eventually cause it to get "stuck" in a 500 error state, even after PHP had returned to idle.
Re:12,000 Requests per second with Nginx, PHP and Memcached by jacques_chester on Tuesday February 15, 2011 @02:13AM (#2220451)
I had a similar experience.

My current setup involves nginx, wp-supercache, php-fpm, apc and mysql query caching. Plus an nginx rule that sets expires headers on all my static files.

It's taken me several years of learning, but now Wordpress doesn't run insanely slow.

Re:12,000 Requests per second with Nginx, PHP and Memcached by haberman on Tuesday February 15, 2011 @03:04AM (#2220545)
Wow, I haven't run my own webserver in a while, but if what you say is true it's remarkable that Apache can be so bad.

If Apache really uses 7-9x more resources than nginx, that speaks more to Apache's awfulness than nginx's awesomeness. You can beat a worthy competitor by 50%, maybe even 100%, but if you're winning by 700% then your competitor is just bad.

I wonder what it does that is so inefficient.

Re:12,000 Requests per second with Nginx, PHP and Memcached by zepolen on Tuesday February 15, 2011 @04:42AM (#2220712)
Given the choice, I'll pick Nginx over Apache any day, however saying that Apache in this case was the problem is just plain FUD.

The OP just had no idea how to set it up properly or made a serious mistake somewhere.

Re:12,000 Requests per second with Nginx, PHP and Memcached by albertzeyer on Tuesday February 15, 2011 @05:06AM (#2220753)
I'm curious. Do you have any thoughts on what he might have setup in the wrong way?
Re:12,000 Requests per second with Nginx, PHP and Memcached by zepolen on Tuesday February 15, 2011 @05:50AM (#2220802)
> I would point out that the server load with nginx running with NO CACHING (WP-SuperCache disabled, all queries execute PHP and perform a MySQL query) was something like 1/4 what my Apache/MPM/FastCGI/PHP/WP-SuperCache-enabled setup was using.

This does not compute - no caching (the use of nginx is irrelevant) will always be slower than caching, especially supercache which creates static files for the server to serve directly - even apache can serve those faster than nginx+memcached of the article.

So I would have to say that his caching system under Apache wasn't working.

That would fit the numbers better.

It is true that nginx will perform better than a badly tuned Apache system in the event of a slashdotting but only because it alleviates the slow client problem ie.:

You have 400 concurrent visitors, each one takes up a thread on Apache causing it to saturate, whereas nginx will buffer the request from the client, send it all in one go to the backend, buffer up the response, freeing the backend for another request - an Apache thread would be unavailable for the entire time of that request.

However you can get the same behaviour just reverse proxying from nginx to Apache.

This is all moot though, you can handle even a slashdotting on a 5$/year vps - just make a static version of the page being hit, and serve it directly until the load dies down. 99.9% of requests will be for that page only.

Re:12,000 Requests per second with Nginx, PHP and Memcached by js2 on Tuesday February 15, 2011 @09:17AM (#2221231)
He might have had keep-alives enabled. Apache + keep-alives + slow clients is a recipe for disaster. Seems everyone learns this the hard way. I learned it the hard way in 1998 when the Starr Report was released and my web farm was running Apache on Ultra 2's.

patio11 learned it the hard way recently too http://www.kalzumeus.com/2010/06/19/running-apache-on-a-memo...

Amusingly, patio11's "Memory-Constrained VPS" have the same amount of memory (512 MBs) as those Ultra 2's I mentioned (I think, they may have actually only has 256 MB at the time).

Re:12,000 Requests per second with Nginx, PHP and Memcached by megaman821 on Tuesday February 15, 2011 @11:52AM (#2221971)
The answer is to put something like Nginx in front of Apache. Let Nginx serve the static resources. Turn keep-alives off and let Apache serve the dynamic stuff. Also disable every module in a Apache that you don't know what it is doing and re-enable them as you understand them to make your site work.
Re:12,000 Requests per second with Nginx, PHP and Memcached by modoc on Tuesday February 15, 2011 @09:31AM (#2221287)
Apache isn't that bad. I haven't used nginx, so I can't speak to it, but the reason I haven't is that I can push 500+ Mbps out of a single box running Apache with just some basic configs. mod_php may be slow, but Apache itself is plenty fast.