No worries at all--glad it turned out to be easily fixable.

And now I know what "nerfed" means. :)

This is why I love HackerNews. Guy asks for SEO help, frigg'n Matt Cutts answers!

FYI - Another fairly common usage of "nerfed" these days (especially in the gaming community) refers to something being toned-down. E.g. if many players are complaining about a character ability being too powerful, the developers may consider "nerfing" that character.

In Internet Marketing, we refer to being delisted as getting "sandboxed"

As you're replying to Google's Web Spam lead, I'm sure he's familiar with the terms for obliterating your splog farms. :)

I have also seen a messed up (404 erroring) robots.txt file cause a site to get deindexed out of the blue

That's a misconception. A 404 on robots.txt will not have any effect on crawling as it's treated the same as an empty robots.txt file allowing all crawling.

But it's different for 5xx HTTP errors for the robots.txt file. As Googlebot is currently configured, it will halt all crawling of the site if the site’s robots.txt file returns a 5xx status code for robots.txt. This crawling block will continue until Googlebot sees an acceptable status code for robots.txt fetches (HTTP 200 or 404).

interesting that needs to go into the webmaster guidelines I was not seeing 500's or having it reported in GWT as errors on the site that it happened to

It doesn't belong in the guidelines but it is described in the relevant section of the Help Center:

http://support.google.com/webmasters/bin/answer.py?hl=en&...

In summary: If for any reason we cannot reach the robots.txt due to an error (e.g a firewall blocking Googlebot or a 5xx error code when fetching) Googlebot stops its crawling and it's reported in Webmaster Tools as a crawl error. That Help Center article above is about the error message shown in Webmaster Tools.

Given that you said you did not see errors being reported, That suggests there was something else going on. If you need more help, our forums are a great place to ask.

Chears I am of on leave for a week ill get this put into our best practice guide for our devs and IS guys when I am back.

Funny thing was I tried resubmitting the main page in GWT an all the traffic came back almost instantly.