HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Cache-Control: private
Connection: close
Cache-Control: max-age=0(I used to telnet to port 80 for testing, and type GET / HTTP/1.0 <enter> <enter>, and that should be LF on Linux & Mac)
Do you ignore whether your HTML is valid just because the browser rendered it correctly?
I've got real work to do. Making a validator happy is fake work.
So, do you want to pay the price upfront when you can plan for it or afterwards when the fix must be done immediately because customers are complaining?
I'd much rather pay the exact price later, than an inflated price now.
Then, working at a startup taught me that it's not black and white. Several quotes come to mind, but Voltaire's is my favorite:
"The perfect is the enemy of the good."
Cowboys may get things "done" quickly, but that doesn't help when things are subtly broken, have interoperability problems, or are nearly impossible to extend without breaking.
If the cache has a copy of an article that is a few hours old it will just give that version to Googlebot while if it thinks a human is requesting the page then it will go to the backend and fetch the latest version.
https://www.varnish-cache.org/lists/pipermail/varnish-misc/2...
[1] 15k reqs/sec on a moderate box
It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then the crawlers would just be pulling out cached, pre-rendered HTML.
There's an example of doing such with nginx here:
http://serverfault.com/questions/30705/how-to-set-up-nginx-a...
With that you'd just have to send out the HTTP header from the arc app saying that current articles expire immediately, and old ones don't.