Re:Ask HN: Please review our startup: historious
by StavrosK on Sunday August 15, 2010 @06:58PM
(#1606196)
We are using Varnish, but we're storing the sources in the DB (to take advantage of automatic compression and all-around ease). It's trivial to get Varnish to cache the entire source and invalidate the cache when the document changes (the public/private setting, basically, so it doesn't accidentally share a page that's been changed to private), so that's all good.
The biggest problem (by far) is disk usage. The rest of the service is very easily sharded, really, as every user is isolated. Solr is also fantastic (much better than Sphinx, in hindsight we should have gone with that for TP), so that can also scale very well (there's even an implementation of it on hadoop).
We'll add the caching feature to the front page as soon as we finish the current round of A/B testing, thanks again (that feature was basically an afterthought, so it was great that you noted its importance)!
Beware and embrace the slashdotting (insert digg, reddit, HN here).
The cache is both your killer feature and your risk. I hope it's statically stored or can be put in memory quickly... are you using Varnish?
Encourage it's use... if people are bookmarking using histori.us and a site goes down, you've just gained a very large audience to your service. But only if your site survives the onslaught itself.