A few weeks ago I got on the front page and within a 24 hour period was hit with 29,000 unique visitors with 38,000 page views. The page itself is image heavy with 1.3 MB on first load. I'm running Wordpress with the Quick Cache plugin by PriMoThemes. I'm hosted on a shared 1and1 server.
I've been hit before and went down, that's when I installed the Quick Cache plugin. Also 1and1 moved me to another server at some point but I'm not sure if it was before or after. Either the cache plugin is really good or I'm on a rockin server all by my lonesome. Or both.
If you're self hosting a wordpress site grab the Hyper Cache plugin or the very very simple Quick Cache plugin by PriMoThemes.
http://www.tutorial9.net/tutorials/web-tutorials/wordpress-c...
Knowing that I can serve 10 requests per second and likely withstand a frontpage on HN is more useful to me than knowing that I need a way out if and when my web-servers are being totally overwhelmed.
http://measuredmass.wordpress.com/2012/10/12/hacker-news/
Graphical post-mortem here:
http://measuredmass.wordpress.com/2012/10/20/more-hn-numbers...
During some large events, I was seeing 5-7 page views per second and Wordpress did just fine (I think I had 120k page views in 24 hours). But I made sure to test as much of my site as possible to make sure it performed well. Using various page analyzers to make sure the proper headers were returned so objects were cached on the client. Tuning wordpress to cache pages properly. Turning on PHP APC (opcode cache). Running various stress tests on the site (apache benchmark and loadimpact).
I'd say the page analysis tools and extended apache-benchmark runs really helped me tune my OS and services properly so I could handle a huge load.
AwStats screen shot: https://unavailable.s3.amazonaws.com/20121130_WP-Super-Cache...
Plug: we've built several products around Varnish so we have a good handle of how/where Varnish can be leveraged. Here's a list of varnish things we've built at unixy:
Varnish load balancer: http://www.unixy.net/advanced-hosting/varnish-load-balancer
Varnish for cPanel and DirectAdmin: http://www.unixy.net/varnish
Varnish w/ Nginx for cPanel: http://www.unixy.net/advanced-hosting/varnish-nginx-cpanel
Email in profile. I'll be more than happy to help out.
Joe
You could come up with a generic VCL that works for most websites out there but its cache effectiveness diminishes as you try to account for the most common corner cases. In fact, we did come up with such VCL. We distribute it with the cPanel Varnish plugin.
If you ever have a question or need a hand with Varnish/VCL drop me an email. I'll be more than happy to help out.
So in theory we were positioned to serve from Nginx cache.
I often try to think of ways to not have full text search...this case is a little more difficult, but why not create a list of common recipe names (i.e. "grilled cheese" would be a facet for all grilled cheese sandwiches) and store it as a static json? It would take some taxonomy work on the backend but the list as JsON could easily be less than a MB and the. You wouldn't have to worry about full search as much...full search could still be an option, just not a top of the front page option.
Then, ensure that Varnish is actually caching every element of the page, and that you are seeing the cache being hit consistently.
You should expect over 10,000 unique visitors within 24 hours, with most coming in the 30 minutes to 2 hours after you've hit the front page on HN.
You need not do your whole site... but definitely ensure that the key landing page can take the strain.
Unless you've put something like Varnish in front of your web servers, there's a good chance your web server is going down, especially if your pages are quite dynamic and require any processing.