I always thought I was being wise by not doing any premature optimization, but after a few lessons learned the hard way I certainly factor in performance to the design of software before I build now.
Scalability is not a "feature" tacked on at the end development.
What percentage of startups on HN have achieved product/market fit, are past a 128GB commodity box AND have no dedicated engineering team for scaling issues?
Once you get to the point of no return you have to know what to do or you'll suffer. Learning that when fire is falling from the skies is the worst way in retrospect.
Plus, almost all of these techniques will also speed up page generation on a lightly-loaded server, so that's a win in any circumstance.
In all of my cases dealing with scale issues it's been due to the DB growing very large in size to the point where you can't use the same techniques that you're used to using. That usually happens in combination with more concurrent users. Depending on how large and how organized your code is, things can get very ugly when the site starts throwing errors, being unresponsive, etc.
I usually know where I want to go as a next step with scaling and monitor our resource usage until it gets pretty high. Then we take the next step. But we don't spend a huge amount of time scaling until we know we need it.
I went back and forth with them about optimizing the app, but it was apparently some huge labyrinthine monstrosity. They insisted that they didn't have the resources to do any of the significant rewrites that it would require to fix the app to do proper queries (or at least, not enough to be worthwhile).
Eventually I gave up. /tmp was mounted onto a separate partition, so I disabled ext3 journalling and set commit=30 so that it only sync'ed to the disk every 30 seconds. Since no temporary tables lasted that long, the VFS layer never wrote to the disk if it didn't have to. /tmp became an in-memory cache, and CPU use dropped to 5%.
Optimizing isn't about a checklist, it's about looking at the system that you have, understanding what it's doing and why, and understanding how the other systems around it behave so that you can resolve the issue. Moving onto another database server wouldn't have helped them. Moving onto a RAID would have reduced the impact, but their load didn't scale linearly so they'd hit their limit in a few months anyway.
Where, in the US, can I rent a 32GB server for less than $100?
[1] http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s
[2] http://www.honelive.com/xml/#new-york-city-dedicated-servers
It's 2012 for god's sake, and 32GB RAM servers are still hundreds of dollars a month to rent from any first class data center.
I'm really curious to see how different equivalently priced ASUS and supermicro boards differ.
Yesterday I had a meeting with a potential customer and I hated it. I hate to try to explain my SaaS software to non-technical people who treat me like some 17-year-old webmaster. I'd much rather be refactoring Clojure code. But I got out of my comfort zone and this client will probably add hundreds of thousands to my bottom line. And I'm glad I was at that meeting while my competitors were fetishizing about non-existent scaling issues.
It's 2012 for god's sake, you can rent a 32GB server for less than $100.