It's nice to see detail like this on high-traffic, high-risk environments.

I'm curious about their provisions for cross-datacenter failover. The article mentions haproxy being ready to direct requests to a different datacenter as well as ELB spanning availability zones. I'd expect a failover option entirely outside AWS as well, with short-TTL DNS ready to make the switch.

I'm also not sure what value varnish brings to the table when the Apaches are just serving a small number of flat files. It seems like unnecessary complexity -- by the same logic found earlier in the article -- when a well-tuned webserver will serve flat files at a comparable rate. Maybe the Apache configuration for this workload was sufficiently different to make it an unwanted risk.

If you look at the current webserving stacks propagating out there, wev'e gone from apache to multi-layered stuff, haproxy/varnish/nginx/apache/mongrel/rails/ various fcgi/etc..... and the thing to keep in mind is there is overlap between most of these projects.

Apache can, one way or the other, do most of what the others can do - possibly not as well, and at the risk of a much more complex configuration.

Haproxy does one thing and does it really,really well - it's great for dealing with load balancing, concurrency limiting per user defined resource, and identifying and routing incoming requests to the right infrastructure. It's really good at this - that's what it does.

Varnish does one thing and does it really, really well - it caches content and serves it up (usually out of memory, but even if it's swapped, it's optimized) to keep the load off your application servers. It has various optimizations built into make it really good at this.

So - in this case, the answer might be "not much - we could just put the apache's out front behind haproxy" - but it appears putting the varnish server out front with a 5 second cache dropped the load on the application servers (in this case apache serving static files that it receives over rsync - dont' forget the rsync part - resources are needed for that). This might result in smoother output for the end user, rather than something hitting a node that's busy servicing an rsync update. It may also be their engineers are very familiar with the haproxy/varnish front end setup, as it presumably exists in their current day to day operation as well.. so the people responsible for keeping things up probably decided "Yes, we'd like to keep it there - it makes our lives easier."

There is an operational anti-pattern in there - removing too many elements from a known system is also a kind of added cmoplexity - all your troubleshooting methods disappear.

Their goal here was to de-couple the dynamic elements from the event-driven side of things and turn them into something more resilient (and less flexible) for a short time to deal with unknown and unpredicably large load.

Doesn't Varnish handle many times more concurrent clients than apache does, at significantly lower system load? It could just be pure optimization.

It does. Apache is strictly one process per connection, varnish handles hundreds, thousands or even tens of thousands of connections with just one process so the overhead is minimal.

Apache has not strictly been one process per connection since 2.0. See the worker MPM. I would have chosen nginx, but Apache can be configured as a capable static file server.

Even then each thread is still one connection (according to the apache docs).

Except in the Event MPM

It is pure optimization - they said it gives them a performance boost and smooths things out.

In their "flat-file" setup though - they are deliberately configured so that, should varnish fail (which was one of their concerns) - they don't actually need it - they could just have haproxy immediateley start hitting the apache servers directly.

So I know nothing about haproxy, but what sort of system load does it generate?

They say they ran it on a EC2 micro instance, and the micro instances are specifically designed to handle spikes, not continuous heavy load. In fact, they intentionally throttle under continuous heavy load.

I guess it must have been the right choice for them, I was just surprised to see it was a micro. Would that typically be the right way to go?

HAProxy is very light. We put it between nginx and our webservers for better failure handling, load-balancing, and some additional logging detail. On a small system (single AMD 1226) it's averaged 3% CPU over the last 15 days while handling ~300M requests.

The micro instance was fronting the render farm. As far as I can tell the article doesn't specify the instance types in use for any of the other components.

Ah! I see that now, thanks for pointing that out!

Ugh. They're not talking about Flat Files. They're talking about HTML. Saying so in the headline would have saved a lot of confusion.

A flat file is essentially a .csv holding data, and can be a fast way to bulk load denormalized data. As such, it actually does have a use in the context of scaling, so it's natural to expect that they were using the term correctly.

A static file, or more simple, a HTML file is what they're actually talking about. As they've noticed, it's what web servers are best at serving, and it scales obnoxiously well.

Now that we're all talking about the same thing, I can say that I've been serving all my product blogs as .html for years, and have never had any of them fall down under load.

It's really easy to get something in place to generate static files from a blog or CMS. I do it with:

- a 404 handler that maps missed requests for .html to their equivalent generator.

- a regular old blog engine that takes an extra parameter "writeThisToHTMLOnceYouveRenderedIt"

Future visitors skip the redirecting and generating and are simply served the static file. Next time you edit the content, you can simply blow away the whatever-blog-entry.html and index.html and know that they'll show up again next time anybody asks for one of them.

> A static file, or more simple, a HTML file is what they're actually talking about.

Static yes, HTML no, an HTML file could still have e.g. SSI instructions. It's valid HTML, but if the webserver supports SSI it's not going to be static.

Well, now we're arguing semantics, but no... an SSI instruction is not part of HTML. It's a server-side scripting language. I could set up my server to interpret PHP embedded in .html files as well, but then they're not really HTML files any more.

Slightly off-topic, but the title of this blog (column?) is amazingly clever:

"All the Code That's Fit to printf()"

Your threshold for amazing is pretty low. :P

Would your impression of the pun be improved if I said that the NYT's motto, on every masthead for over a century now, has been "All The News That's Fit To Print"?

I understood the joke perfectly, it's just not that witty...

I approve of this level of paranoia: It's never failed, not even once but just in case...

I have always been a proponent of the 5 second cache for things like blog posts. After you hit Reddit, you can see 1000 requests a second for a few minutes. If you cache for 5 seconds, nothing is ever very far out of date, but you save yourself 5000 requests. Seems like a win-win.

(I'm also a big fan of Varnish. I tried it out this weekend on a site that basically serves an HTML file that says "hello world". Apache can do about 11,000 requests a second, but Varnish can serve 15,000 requests a second. Excellent!)

Chances are that you will need to do some serious kernel tuning before you will really run in to the limits of what varnish can do.

If you're using linux using IPV4 have a look at the following parameters in /proc/sys/net/ipv4:

tcp_tw_recycle

tcp_tw_reuse

Tuning those will help in allowing faster re-use of sockets in the TIME_WAIT state. This matters because at the defaults your sockets will linger for a long time before being allowed to be re-used (as per the RFC). Technically this is the correct behaviour but it can quickly become a bottle-neck.

Ulimit max open files per process:

ulimit -n 50000

The default is just 1,024, and that's not nearly enough to keep varnish working hard.

Those are the first things to look at, there are many more once you start to bottom out again, but this will get you started.

A good way to see if you've got your kernel tuned properly is when varnish starts to approach the limits of your hardware, I've seen it do well over 600Mbps on an otherwise unloaded box.

Why stop at 5 seconds? Unless you're updating that blog entry every 6 seconds, you're missing out on a bunch of good caching by regenerating it 12 times a minute.

How about cache it once at write-time, then again whenever you edit it?

Rather than repopulating the same cache entries every 5 seconds you could instead actively purge the Varnish cache entry when the content actually changes.

You get 1000 requests a second from Reddit? Wow, it's a whole lot bigger than I thought.

Now that I am remembering better, I actually hit the front page of Reddit and del.icio.us at the same time. Those both drive a surprising amount of traffic. (The article was "Git merging by example", now archived here: http://blog.jrock.us/posts/Git%20merging%20by%20example.pod)

I did not use varnish at the time, but I basically kept up with the load. My pages were cached with an in-memory cache in the app layer, which worked well enough, I guess. As blog.jrock.us mentions, I was unhappy with the design of my software, so I took it down. Two years later, I almost know what a good design is, and so I should have a blog again soon. But I digress :)

Needs to prove it!

This is exactly why I created StaticGenerator for Django: https://github.com/luckythetourist/staticgenerator

On the drupal side there is Boost: http://drupal.org/project/boost It has some fairly smart cache invalidation logic as well. At work we use the cache invalidation logic to tell varnish what needs to be refreshed. We use this setup for over 1,000 TV station news sites; supported by 10 boxes total. Flat file cache size is around 80GB. Our load is usually under 2 on all boxes.

Forgive my ignorance, but does "flat file" mean "prerendered static HTML page" in this usage?

Sounds very much like it. From the article:

>After each batch of new data was received from the AP, this server determined which pages needed to be re-rendered and, using the Typhoeus libcurl-multi bindings for Ruby, pulled new data for each of these pages from the render pool.

Sounds like their infrastructure already has a farm of servers set up to handle rendering articles into their HTML components, so for this scenario they would save the output as a static file and then write it to disk rather than sending the HTML out to the response / caching layer / etc.

All righty... it was just a usage of the term "flat file" I was unfamiliar with. :-)

Yeah, I think they made that up. I've only ever encountered the term "flat file" in the sense of a flat file database, such as a CSV file. As you say, pre-rendered static page is a much better name for it.

I have always used the term "static html" (as opposed to "dynamically generated html").

"Flattened files" may have been what they were meaning. "Flattened Files" would be a nice simple term for them if there wasn't confusion with Flat-file databases.

Ah! Now it makes more sense, etymology-wise and all that.

Yes. The diagram explains that their render farm takes data from AP and converts them into pages that are then sync'd with apache.

the filesystem is just another datastore. using it like this means you're spreading out the requests per second across each individual server's available IO, however you've also forsaken the "getting data from point a to point b" features of other datastores and therefore have to do it yourself (usually rsync).

to be honest, since there wasn't actually a problem to solve as varnish is setup as both an HA environment and to use the grace/saint features, this is a case of overengineering in my book.

When dealing with unpredictably high traffic spikes, on days when you cannot afford any kind of downtime - when that downtime comes at great cost, it's not overengineering (and it's hard to determine if it's overengineered until you know the budget and time spent on the actual project - this may have been a relatively simple modification all things considered)

Static content is easy to crank up to web-scale. You can use DNS, any kind of load balancer, all kinds of web services, CDNs, whatever.

Dynamic content is hard to scale (compared to static). You have several layers of added complexity. Yes, you can build wonderful, self-scaleable systems - but at some point they hit a limit, there are many more resources that can be tied up, and troubleshooting and scaling that out beyond anything you've previously imagined on short notice can take time you can't afford.

So - simply de-coupling the dynamic content generation from the static web serving is a great way to make a clean break - you now have a known & tuneable load on your dynamic application (because your'e running it at known intervals, rather than being event driven by user requests) and you have a front-end static infrastructure that you can scale like mad, and even if your back-end collapses, edit by hand.

Surely there are other ways to approach the problem.... but it also depends on the engineers involved, the time taken, and their confidence in their ability to deal with it.

I'm also fairly sure they aren't the first company out there to take this approach to burst scalability issues... but it's curious to note how the NYT actually operates.

TL;DR: Look at the old configuration, and the new configuration. Decide which one will best serve your business in terms of your ability to troubleshoot it when it gets hit by a level of traffic higher than you can plan for, because you have NO idea how high it will go.

In my book, overengineering is defined as doing more work than is necessary to ensure that the risks are within acceptable bounds.

As the author points out, having the Times election site go down on election eve would be a BIG PROBLEM -- massive losses in both reputation and advertising revenue. For the system architect, a failure could possibly mean losing his job.

What the author has laid out is a system that is robust to multiple, simultaneous failures (with possible exception of the loss of AWS, although that's not entirely clear). That just seems like good planning.

The file-system isn't just another data-store: it's I/O and therefore special. On systems with 'sendfile' data can be sent straight from disk to network port without CPU time (and any copies being made). This is used extensively by high-performance webservers and web-accelerators.

sendfile() still requires CPU time - it just passes the job of sending the data over the socket to the kernel to finish up. The previous method would have been a select() loop or similar that ensured the data was sent out to the kernel, involving a bunch of system calls and context swtiches. sendfile lets you hand the job off 100% to the kernel and have your thread move on.

I was under the (possibly mistaken) impression that Direct Memory Access (DMA) hardware could move data straight from the Disk Cache to the Network card without touching the CPU Caches at all.

True - DMA can do that - but then you'd have to have something managing the network stack - the kernel manages tcp/ip.

From the man page:

"sendfile() copies data between one file descriptor and another. Because this copying is done within the kernel, sendfile() is more efficient than the combination of read(2) and write(2), which would require transferring data to and from user space."

I may sound naive here, but: if all you're doing is serving just 184 flat files, then why do you need all this RoR jazz? Can't a bunch of Apache/Nginx servers behind a load balancer (Varnish, TrafficServer, etc.) handle that just fine? Throw in a DNS-based scheme for failover/redundancy?

Ruby on Rails is generating the flat files, while apache et al. are serving them.

It's risk mitigation... this is one way you can deal with unpredictably large spikes in traffic.... for the NYT - elections probably represent their extreme.

For the regular ebb and flow of data, you can plan, and use varnish/rails/nginx/ all the magic tools you want, and things will work great - but you also have added complexity, and when something goes wrong, the more complex the system, the longer it generally takes to fix. Especially under unexpected heavy load.

Simplifying the system down to flat files and de-coupling the ROR stuff gives you a clear troubleshooting point - if something goes berzerk, you can cut the link between the two and troubleshoot in relative safety before turning replication back on.

I'm rambling - the main point is to reduce the complexity involved in the end-user transaction to be as efficient and fast as possible so you can deal with an unknown load factor coming in on a really important day. Going down that day would be BAD for business.

The load was for election night, which is more than likely a full order of magnitude, or two above their normal traffic levels. So their normal setup of RoR is fine for normal amounts of load, but would fail under that expected spike. So they used all their page-creation code in rails, but instead of serving it out like normal, they create the static files, and serve those. You lose out on interactivity, but make up for it in performance.

> I may sound naive here, but: if all you're doing is serving just 184 flat files, then why do you need all this RoR jazz?

Generate the static files: those files contained live election results, so they had to be regenerated as new results came in.

I'm really curious now how many other large setups use a similar setup (memchache, varnish, apache, rails) for their sites... anyone know a good sites that's put together any statitstic?