As much as it sucks when Github goes down, there's a sense in which I look forward to it happening. It's yet another chance for me to get a peek at their architecture. As a high school student it's pretty disappointing to me how hard it is to find fully laid out architectural plans for large websites. So I appreciate sites like Github and Ravelry being open with that information.

Google architecture: http://highscalability.com/google-architecture

Google infrastructure: http://en.wikipedia.org/wiki/Google_platform

Amazon architecture: http://highscalability.com/amazon-architecture

No one will layout the blueprint for you, but most general information regarding infrastructure are out there.

Wow, thanks for bringing highscalability.com to my attention. Awesome breakdown of sites.

Wow, badass comment. I had no idea, thanks!!

highscalability.com == architecture/infrastructure porn.

Another Yahoo! bot behaving badly? Between them and MSN/Bing's bot, what the hell is going on with search engine spiders lately?

Imagine you have to write a computer program to visit every page of every website on the entire Internet.

Sometimes you are going to make a mistake, and someone on the Internet is going to notice.

Except this has been going on for months, has happened a number of times, and occurs in violation of standards, like robots.txt, that were created to prevent it from happening. I'd expect it from an individual or startup, but not a large company that's been doing spidering for a long time.

From the explanation it looks like github isn't using '5 whys' when things go wrong. Has anyone tried this? Is it overkill or does it work?

http://en.wikipedia.org/wiki/5_Whys

If I were GitHub I think I would have mentioned the cause of outage number 3 after I put the rate limiting fixes in place, which is not the impression I got from the article. It is not hard to write a basic HTTP request spoofer and it sounds like they're still vulnerable.