User-agent string changes

Re:User-agent string changes by userbinator on Saturday November 22, 2014 @11:43AM (#8645913)

Absolutely. I use a filtering proxy which among other things enables modifying request headers, and I've experimented with a ton of different UA headers over the years; and it's astounding how many sites will fail in mysterious ways if your UA doesn't match one of the ones they're expecting (perhaps this has decreased a bit now with all the diversity in browsers.)

In particular, I've noticed that having absolutely no UA header produces the most horrible results: "500 Internal Server Error"s, "we think you are a bot so you have been denied access" (403s are common), and even IP bans.

UA strings were intended only as a form of branding/advisory, but I think they're now being used in an unfairly discriminatory way. Denying access because of a "foreign" UA is almost like racial profiling.

[ Parent | on HN ]

Re:User-agent string changes by sciurus on Saturday November 22, 2014 @12:21PM (#8646027)

I don't think your analogy works.

Racial profiling is when law enforcement decides to target someone someone based on their ethnicity (an innate characteristic) instead of their behavior.

Whether or not an HTTP client sends a user-agent header is a part of its behavior when making a request. Its fair to observe that along with other aspects of the clients behavior when deciding how to handle it.

You're probably being blocked by sites running WAFs like ModSecurity. There's a lot of crap out there, and I don't blame people for setting a low threshold for blocking unusual requests. Here's the rules which would catch a missing user-agent: https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/mas...

[ Parent | on HN ]