(Current web crawler problem: sites that won't let you read their robots.txt file if they don't like your user-agent string.)
That's hilarious. So do you use borrow a browser's user-agent or do you ignore the robots.txt?
That would get them to fix the issue pretty quickly :-)
More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.
they told you this because they probably fixed the bug in a newer release. what response were you expecting, exactly? to dive in with a hex editor?
for what it's worth we run into the same issues with our network devices, it's a pain in the ass, and that's why we're shifting to SDN.
"You've obviously put a lot of time in to this. Thanks. We actually had found that bug 4 months ago in R1892, and it's patch in R1899. Please upgrade, or let us know that you're, in fact, using R1899 and still experiencing the issue (it's issue #847 on our bug tracker at foobugz.xyz)"
Or maybe...
"You've obviously put a lot of time in to this. Thanks. We actually had found that bug 4 months ago in R1892, and we're planning on a maintenance release next quarter. If you'd like to test out our upgrade version, please contact [email protected] and let them know you want to be in the testing group - he'll get you the appropriate documentation and code. Thanks for helping fix and test these issues!"
"Update your software" is often a BS answer that can introduce many other issues or breakages in production systems. Without a clean "downgrade" path (which many companies don't provide) you run a risk of introducing more problems.
in other words, what i'm saying is to expect that kind of response is naive, and that you should seek alternatives.
Yes. My favorite was a Coyote Point load balancer bug. If the last character of the HTTP options is "m", the connection will not get past the load balancer.[1] I found this because a web crawler was having trouble with one site. Fortunately, I knew someone with their own Coyote Point load balancer, and was able to establish that the connection went into the load balancer and never came out.
The load balancer has a big file of rules which contain regular expressions. Somewhere, I think there's a "\m" where they meant "\n". Reporting this to the vendor, along with a Python program to demonstrate the problem, was of course futile; they suggested "upgrading the software". I demonstrated that the bug existed on their own load balancer on their own site. I finally added a completely useless field to the HTTP header so that the last character was not "m".
[1] https://www.webmasterworld.com/webmaster_hardware/3312997.ht...