Disqus: Scaling the World’s Largest Django Application

Re:Disqus: Scaling the World’s Largest Django Application by moe on Tuesday February 22, 2011 @11:46AM (#2250399)

On the upside, there's long phases of "just add more of the same" in scaling these things. The fun comes in waves, every time when you hit one of the various physical barriers (namely latency and bandwidth).

That is to say the host-count in isolation is not the most interesting figure. I've seen large sites run on 20 machines or on 500, depending on the skills of the management- and developer-team, and how much they care about the infrastructure cost in the big picture.

The host-count becomes more interesting when you relate it to the request rate. 17k/sec is absolutely a worthwhile workload, even when (as likely in the disqus case) reads dominate writes by far.

That said the relation of 100 hosts / 17k rps seems about reasonable.

However (not meaning to narrow their achievement) the engineer in me can't help but wonder if perhaps even a little more could be squeezed out on the caching front? I was a bit surprised to not see varnish on the slides; fragment caching on the perimeter can achieve mind-boggling results.

[ Parent | on HN ]

Re:Disqus: Scaling the World’s Largest Django Application by thedz on Tuesday February 22, 2011 @01:21PM (#2250878)

Varnish wasn't in production at that point. We're testing/using Varnish now for some things. It definitely is helping.

In the general caching front though, I want to note though that Disqus is particularly hard to cache -- there's a very long tail leading to relatively a high miss:hit ratio per pound of caching.

[ Parent | on HN ]

Re:Disqus: Scaling the World’s Largest Django Application by sugarcode on Tuesday February 22, 2011 @01:21PM (#2250883)

Great points. We've actually deployed additional caching on the frontend, and Varnish should be appearing on future slides.

[ Parent | on HN ]