News for hackers. Stuff that matters.
Re:12,000 Requests per second with Nginx, PHP and Memcached by edanm on Tuesday February 15, 2011 @02:48AM (#2220515)
A question I hope someone here can answer:

I'm now working on my first major webapp, so I've been learning how to be a sysadmin, pretty much from scratch (almost no previous experience with running webapps, lots of experience with other things though).

Anyway, I've been trying to find resources to help learn, but it's been extremely difficult. These are the questions that are still bugging me, hopefully someone here can point me to some good reading about them. Note that my stack is Python/Django over Apache (mod_wsgi) right now. Also note that my site has a "ping" system, where each client connected continuously pings the server every few seconds to see if any new info arrived, making me have to handle a larger amount of requests per second, I believe.

The questions:

1. What kind of load do I need to handle? Is 12,000 requests per second terrible/good/great performance? How do I go about figuring out how many people are online at once for most sites? How do I even estimate it?

2. How can I test the performance of my application? I've learned that Apache Bench is used a lot, but are there better tools?

3. What are the best tools to help me monitor and understand the load on my server?

4. How do I go about understanding the bottlenecks in my application? Right now, my Apache process is taking most of the cpu. What does that imply about where I should optimize?

Sorry to braindump, but I've been looking for answers to these questions online and haven't found any clear help.

Re:12,000 Requests per second with Nginx, PHP and Memcached by jacques_chester on Tuesday February 15, 2011 @03:39AM (#2220606)
As a "meta-answer", you might consider breaking these questions up and asking them on one of the stackexchange sites.

However:

1. "What kind of load" depends on your anticipated visits to the site. There's no substitute for real traffic, but simulation can do in a pinch. Likewise there's no substitute for simulation, but capacity planning tools can do in a pinch. Look at The Art of Capacity Planning, The Art of Application Performance Testing and Guerilla Capacity Planning.

2. There are many tools, some quite sophisticated. Look at siege, httperf and others like them. Consider also using automatic UAT tools (like Cucumber) as the basis for performance and stress tests.

3. Look into monit, zenoss or nagios. There are others.

4. Profiling, profiling, profiling. And ditch Apache, it's a memory hog. Consider lighttpd or nginx.

Re:12,000 Requests per second with Nginx, PHP and Memcached by edanm on Tuesday February 15, 2011 @04:05AM (#2220635)
Thanks, I'll look into those.

When you say ditch Apache, do you mean completely? I know a fairly common setup is to use nginx+Apache to serve Django (with nginx serving the static content and Apache the Django code). Should I be running everything from nginx?

Re:12,000 Requests per second with Nginx, PHP and Memcached by jacques_chester on Tuesday February 15, 2011 @04:09AM (#2220643)
Personally I don't see the point in keeping Apache. It just adds more moving parts and more points of failure to have two HTTP daemons.

I'd ditch it entirely and switch to WSGI or FastCGI.

Re:12,000 Requests per second with Nginx, PHP and Memcached by 2mur on Tuesday February 15, 2011 @11:53AM (#2221980)
You should at least try nginx/gunicorn (http://gunicorn.org/). And if your clients are constantly pinging and the data hasn't changed, make sure you are caching it in memcache (etc) to relieve the database.

http://ericholscher.com/blog/2010/aug/16/lessons-learned-das...

Re:12,000 Requests per second with Nginx, PHP and Memcached by edanm on Tuesday February 15, 2011 @12:04PM (#2222043)
Honestly, I'm a little scared of using something like gunicorn. I could be wrong, but it seems like an extremely new technology, especially when compared to Apache, which has a lot of history behind it.
Re:12,000 Requests per second with Nginx, PHP and Memcached by antihero on Tuesday February 15, 2011 @11:59PM (#2225075)
You can have a failover, so if one breaks you can quickly boot the apache one back up.
Re:12,000 Requests per second with Nginx, PHP and Memcached by zeemonkee on Tuesday February 15, 2011 @04:18AM (#2220657)
With Django - or any ORM it's a good idea to keep an eye on the SQL that your application is running. Use the django debug toolbar[1] and run your SQL queries against a profiler. Most of the time, that's where your serious bottlenecks will be. If a query happens to be a bottleneck, there are ways the ORM can improve performance (e.g. select_related() to reduce the number of queries in a loop, only() and defer() to reduce the number of columns in a query, and so on[2]. In some cases this may even involve using pure SQL or caching heavily used queries.

Look at using processes vs threads with mod_wsgi, which should help improve performance. Ultimately however you might find nginx a better option, particularly if you are using it for static media/load balancing anyway. I'd suggest uwsgi (also heard good things about gunicorn). Use supervisord or another process monitor to make sure the lights stay on.

[1] https://github.com/robhudson/django-debug-toolbar [2] http://docs.djangoproject.com/en/dev/ref/models/querysets

Re:12,000 Requests per second with Nginx, PHP and Memcached by roel_v on Tuesday February 15, 2011 @05:17AM (#2220764)
"Is 12,000 requests per second terrible/good/great performance?"

Re: this, I don't understand this type of benchmark. A 'request' is completely undefined. Serving 12k 5-byte static pages is quite different from serving 12k frontpages with 25 db queries behind them. Is there an industry-standard interpretation of 'request' that I'm not aware of?

Re:12,000 Requests per second with Nginx, PHP and Memcached by fedd on Tuesday February 15, 2011 @07:34AM (#2220969)
if the guy re-formulates his question like

"is 12000 requests the norm for a popular web site"

would it be better

Re:12,000 Requests per second with Nginx, PHP and Memcached by Mikushi on Tuesday February 15, 2011 @03:08PM (#2223025)
No.
Re:12,000 Requests per second with Nginx, PHP and Memcached by fedd on Wednesday February 16, 2011 @02:49AM (#2225572)
damn.

i'm happy i have no questions! :)

Re:12,000 Requests per second with Nginx, PHP and Memcached by Mikushi on Wednesday February 16, 2011 @09:55AM (#2226556)
I realize now that my "No." was misplaced, my "No" was the answer to "is 12,000 req/sec normal for a popular website".