I estimated >50% of our traffic was from search engine crawlers specifically (billions of requests per month) way back in 2010. It's only gotten worse since then. http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh...