News for hackers. Stuff that matters.

Hire YC: Need high-traffic Django help ASAP

hey,

we just re-launched out site w/ Django, and are cant handle the peak traffic load. we have apache in the back (mod_python), nginx in the front, memcached. the database doesn't seem to be the bottleneck.

restarting apache yields fast response times up front, but gradually drifts to about 6-12 seconds per request.

we can't handle getting crushed with unacceptable load times much more without serious consequences, so if anyone is available to help, email me at `mattdennewitz at gmail` and lets talk.

( 8 comments | 6 points | on HN )
Re:Hire YC: Need high-traffic Django help ASAP by maney on Thursday March 12, 2009 @09:45AM (#513411)
The first thing you want to do is measure the problem in enough detail to identify the cause. I can recommend Rob's Debug Toolbar (though with reservations about the possible overhead, though I'd have been happy to chance that when I needed to instrument an underperforming site):

http://github.com/robhudson/django-debug-toolbar/

though his tree doesn't have the injectable measurement points I've added to the timer panel yet. I hesitate to suggest this for a production setting only because it's not had much third party exposure (or if it has no one's let me know), but "it works for me":

http://github.com/mmaney/django-debug-toolbar/

You might want to avoid the most recent commit (so use da1429...) - Rob had asked about adding support for the cache API's "add" method, and I threw that together based on the spec; that one part is completely untested other than that it doesn't cause a compilation error in my test setting.

Re:Hire YC: Need high-traffic Django help ASAP by suhail on Wednesday March 11, 2009 @09:28PM (#512816)
buy a server load balance with nginx and throw a duplicate copy of everything on that and use the same db since it's not your bottleneck.

Start looking for code you can optimize such as bad loops.

Re:Hire YC: Need high-traffic Django help ASAP by chiffonade on Wednesday March 11, 2009 @10:13PM (#512877)
... why can't you just add more application servers?
Re:Hire YC: Need high-traffic Django help ASAP by erlanger on Wednesday March 11, 2009 @06:49PM (#512763)
mod_wsgi. If nginx is serving as a proxy or a file server too? If the former, I'd go with HAProxy. If the latter, Lighttpd works very well with Python and FastCGI.
Re:Hire YC: Need high-traffic Django help ASAP by mattdennewitz on Wednesday March 11, 2009 @06:54PM (#512773)
just switched to modwsgi

response time is up to 30 sec with 1 thread and 50 processes. nginx is up front as a proxy and a file server.

Re:Hire YC: Need high-traffic Django help ASAP by erlanger on Wednesday March 11, 2009 @06:56PM (#512776)
Have you benchmarked the same app running on Lighty and FCGI? You shouldn't have to change much, if anything.
Re:Hire YC: Need high-traffic Django help ASAP by mattdennewitz on Wednesday March 11, 2009 @06:57PM (#512780)
no. is that a common solution for hi-traffic service?

what would lighty/fcgi replace?

Re:Hire YC: Need high-traffic Django help ASAP by a_toker on Wednesday March 11, 2009 @10:11PM (#512874)
Lighttpd would serve both static files and the application. Reddit uses web.py with Lighty+FCGI, and I see no reason that you couldn't run your Django app on it. I use Apache + mod_wsgi for the Apache modules, but I'm taking a good look at lighttpd+FCGI to get more out of my server for my Django and Trac installations.