Just to confirm what I said elsewhere, this site doesn't have any manual spam actions or anything like that. It's just a matter of Google trying to pick the correct canonical url when you have a lot of different (www, non-www, http, https) urls you're showing. If you make things more consistent, I think Google will stabilize on your preferred url pretty quickly.
And now I know what "nerfed" means. :)
But it's different for 5xx HTTP errors for the robots.txt file. As Googlebot is currently configured, it will halt all crawling of the site if the site’s robots.txt file returns a 5xx status code for robots.txt. This crawling block will continue until Googlebot sees an acceptable status code for robots.txt fetches (HTTP 200 or 404).
http://support.google.com/webmasters/bin/answer.py?hl=en&...
In summary: If for any reason we cannot reach the robots.txt due to an error (e.g a firewall blocking Googlebot or a 5xx error code when fetching) Googlebot stops its crawling and it's reported in Webmaster Tools as a crawl error. That Help Center article above is about the error message shown in Webmaster Tools.
Given that you said you did not see errors being reported, That suggests there was something else going on. If you need more help, our forums are a great place to ask.
Funny thing was I tried resubmitting the main page in GWT an all the traffic came back almost instantly.
Would "http:www.apple.com", "https:www.apple.com", "http:apple.com" and "https:apple.com" be treated by Google as four completely different and separate sites also to be ranked in isolation of each other? Why?
Many sites, for example give users their own "name.whatever.com" subdomain. In those cases treating the sites as the same doesn't make any sense.
Second, in webmaster tools, should we always have the www and non-www setup so we can do the "change of address". For example, if www.mysite.com is my preferred URL, do I need to also make sure mysite.com is in webmaster tools and change the address to go to www.mysite.com?
Obviously we can't be everywhere and we can't answer every question, but we try as much as possible to help when we can.
Those folks would have spotted these issues pretty quickly.
For example, if http://mydomain.com/path and https://www.mydomain.com/path have 95% content correlation and repeated requests to http://mydomain.com/path have 95% content correlation, and the server headers look the same, why would it not be safe to decide those are duplicates of a single canonical url?
It's not safe to merge www.domain1.com and www.domain2.com. it's not safe to merge subdomain.domain.com and www.domain.com. However, for the limited cases of www and no-www, https and http, if they look similar, I think it's harmful not to treat them as the same site. You can't expect every website owner to be aware of this issue.
If it's a matter of not being able to be 100% sure, is there a single site that cares about google ranking that runs different sites on different combinations of www/no-www and https/http, but has similar content that would confuse a simple heuristic looking at page similarity? In what sort of circumstance could that happen other than with placeholder pages?
GWT allows selecting a preference between www and no-www, but I don't see a preference between https and http. I think Google should add a notice that using GWT to select between www and no-www is deprecated and the recommended way to handle www, no-www, http, and https selection is to use 301 redirects or rel="canonical" tags.
I work at Google helping webmasters like this.
As far as I can tell, there are a couple of interacting issues we're seeing on the site that can be causing what you're seeing. It's a bit technical, but it's easy for you to implement a fix.
Firstly, our algorithms recently have been picking one of the following URLs as the canonical URL for the homepage:
http://www.safeshepherd.com/ https://www.safeshepherd.com/ https://safeshepherd.com/
For example, I see that the non-HTTPS pages redirect to the HTTPS pages (e.g. http://www.safeshepherd.com/ to https://www.safeshepherd.com/), but the non-www pages do not redirect to the www pages (both https://www.safeshepherd.com/ and https://safeshepherd.com/ return content). When we find the same content on multiple URLs like this, our algorithms pick one representative URL, and over the past few weeks the choice has been changing. As of 3 days ago, the current choice is https://safeshepherd.com/ .
As it stands, our algorithms are trying to figure out the right canonical URL format, but it's difficult in this kind of situation. You can help by redirecting to your preferred URL format (say https://www.safeshepherd.com/*), and our systems will pick up this signal, and that will be reflected in the search results and reporting in Webmaster Tools.
Secondly, Webmaster Tools treats these as different sites. For example, you would need to verify and check the statistics of both https://www.safeshepherd.com/ and https://safeshepherd.com/ (as well as the HTTP versions) as they're separate sites. It may be that you're checking (say) the stats for http://www.safeshepherd.com/ but if our algos have picked the https://www.safeshepherd.com/ URLs as canonical, the search queries of the former will suddenly be closer to zero but the latter will be a more accurate reflection of the site's traffic.
Hope this helps, Pierre