However:
1. "What kind of load" depends on your anticipated visits to the site. There's no substitute for real traffic, but simulation can do in a pinch. Likewise there's no substitute for simulation, but capacity planning tools can do in a pinch. Look at The Art of Capacity Planning, The Art of Application Performance Testing and Guerilla Capacity Planning.
2. There are many tools, some quite sophisticated. Look at siege, httperf and others like them. Consider also using automatic UAT tools (like Cucumber) as the basis for performance and stress tests.
3. Look into monit, zenoss or nagios. There are others.
4. Profiling, profiling, profiling. And ditch Apache, it's a memory hog. Consider lighttpd or nginx.
When you say ditch Apache, do you mean completely? I know a fairly common setup is to use nginx+Apache to serve Django (with nginx serving the static content and Apache the Django code). Should I be running everything from nginx?
I'd ditch it entirely and switch to WSGI or FastCGI.
http://ericholscher.com/blog/2010/aug/16/lessons-learned-das...
Look at using processes vs threads with mod_wsgi, which should help improve performance. Ultimately however you might find nginx a better option, particularly if you are using it for static media/load balancing anyway. I'd suggest uwsgi (also heard good things about gunicorn). Use supervisord or another process monitor to make sure the lights stay on.
[1] https://github.com/robhudson/django-debug-toolbar [2] http://docs.djangoproject.com/en/dev/ref/models/querysets
Re: this, I don't understand this type of benchmark. A 'request' is completely undefined. Serving 12k 5-byte static pages is quite different from serving 12k frontpages with 25 db queries behind them. Is there an industry-standard interpretation of 'request' that I'm not aware of?
"is 12000 requests the norm for a popular web site"
would it be better
I'm now working on my first major webapp, so I've been learning how to be a sysadmin, pretty much from scratch (almost no previous experience with running webapps, lots of experience with other things though).
Anyway, I've been trying to find resources to help learn, but it's been extremely difficult. These are the questions that are still bugging me, hopefully someone here can point me to some good reading about them. Note that my stack is Python/Django over Apache (mod_wsgi) right now. Also note that my site has a "ping" system, where each client connected continuously pings the server every few seconds to see if any new info arrived, making me have to handle a larger amount of requests per second, I believe.
The questions:
1. What kind of load do I need to handle? Is 12,000 requests per second terrible/good/great performance? How do I go about figuring out how many people are online at once for most sites? How do I even estimate it?
2. How can I test the performance of my application? I've learned that Apache Bench is used a lot, but are there better tools?
3. What are the best tools to help me monitor and understand the load on my server?
4. How do I go about understanding the bottlenecks in my application? Right now, my Apache process is taking most of the cpu. What does that imply about where I should optimize?
Sorry to braindump, but I've been looking for answers to these questions online and haven't found any clear help.