Goodness me how I could rant endlessly on this subject.

I operate an automated web frontend testing service and much of that centres around retrieving a HTML document and running some tests against it.

I have tried very hard to be nice and fair and to set appropriate UA strings, such as featuring only the product name and relevant version numbers. Unfortunately for reasons relating to how responses are altered in relation to the UA string this is not possible.

My product features the word 'test' in the name. Some server-side services return a 404 or a 500 if the UA string contains 'test' in any form. Due to this I can't include the full product name in the UA string and expect all tests for all end users to work in cases where they really should. Some others respond similarly is the UA string is only 'agent'.

The number of services that respond in a different manner to a blank UA string is significant. Likewise for cases where the UA string is not somewhat similar to that of common browsers.

On a related subject, I'd love it if everyone supported the simple HEAD method consistently.

Some services respond as expected and return only the response headers. Some services respond fairly with either a '405 Method Not Allowed' or '501 Not Implemented', giving me the option to try again with an equivalent GET request. Some services send a 404 or 500 in response to a HEAD in cases where the equivalent GET request works just fine.

And lastly, https://myspace.com/ responds with nothing when making a HEAD request and you have to wait for the request to time out in cases where an equivalent GET works just fine.