The user agent header should have been dropped a long time ago, as it is one of the largest[1] sources of browser identification data.

Even ignoring the tracking problem, using the user agent string instead of feature detection has always been terrible engineering, as it assumes all browsers within a version have the same features. Browsers have always been configurable - often in extreme ways. I don't understand how anybody can simply6 pretend the preferences dialog (and extensions, etc) doesn't exist.

[1] Out of the 15.91 bits of data that Panopticlick was able to collect from my browser, 9.0 bits (56%) were from the user agent. The 2nd largest source is HTTP_ACCEPT at 4.25 bits.

Feature detection is not always enough. E.g. if a browser reports it has a required feature but that feature is buggy rendering it effectively unusable, then browser detection is necessary.

> using the user agent string instead of feature detection has always been terrible engineering

Is there a better solution for server-side feature detection?

Something like the Accept header, listing capabilities. Or something like how Android handles capabilities in manifests.

BTW, feature detection can be even more identifying (although you need it anyway). IIRC my browser was completely unique at Panopticlick, most likely because I have a rare plugin installed (a handler for byond:// URLs)

Or, make the information in the string useful. Like, maybe, just list the features the particular user-agent supports.

User-Agent: entire browser source code (uuencoded) ?

Hmm... might be a bit big... how about, since the source code is publicly available, we could just put the browser name and version in there?

No, seriously. Can we just get all browser vendors cooperate and clean up the mess at the same time, so everyone wins?

User-Agent: Chromium/47.0 (Windows 10)

User-Agent: Firefox/43.0 (GNU/Linux; X11)

User-Agent: Wget/1.15.4 (GNU/Linux)

etc.

Given that current UA strings are abysmal (and sent on every request), this would probably save a few kilobytes worth of headers (images, scripts, stylesheets), so would probably improve performance by some nanoseconds.

They managed to do this with SHA-1 deprecation and a lot of other stuff. So they could just announce that "on February 1st, 2017 we all clean up the clusterfuck User-Agent headers are", bake in the timer and spread the word.

What's the cost/benefit on that? Older sites would not update. Newer sites are using better forms of detection, except for the tiny number of sites that put engineering effort into supporting an insane long tail of old browsers.

(SHA-1 deprecation only worked because certificates have to be renewed, so any site with working HTTPS is being actively maintained at some level)

Benefit's obvious - getting rid of legacy cruft. It has to be done someday, or by 2030 we'll end up with few kilobyte-long UA strings.

Cost: bajillion man-hours of CEOs, companies and committees trying to negotiate that they all do this at once, then about 10 minutes of developer time to actually code this into reality.

Then some sites would be broken, but old stuff eventually rotting and dying is nothing new on the web. Unfortunately, no idea about the numbers, but I suppose it would be low. If someone has the actual statistics on how much sites become practically unusable without a legacy User-Agent header, that'd be great.

> Benefit's obvious - getting rid of legacy cruft. It has to be done someday, or by 2030 we'll end up with few kilobyte-long UA strings.

Which will cost how much by the standards of 2030? Particularly given HTTP/2 header compression.

Getting lower response times isn't a main point here. Getting rid of the cruft is about sanity, not extra milliseconds.

But I'm sure even by 2040 there still will be a lot of places with awful network connectivity, where every kilobyte would still matter.

If you're going to go from this "Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; BLU; WIN HD W510u) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537", to something useful, you should probably give the useful one a new name and drop u-a on the drop dead date... That way websites can test the new way with their audience in advance.

The obvious solution is to deprecate User-Agent and have a new header. Browser-Version or something like that.

some stores even charge you a different amount based on user-agent string...

How do they encourage user tracking? The user agent string only tells you information about the browser, not the user.

It's not the only data that is available for deriving a tracking key.

https://panopticlick.eff.org/

Why are you pretending that there is some sort of difference between "tracking a user" and "tracking the user's browser", which is close enough for most tracking purposes? Tracking the browser is tracking a user for a vast majority of people. For browsers on a phone, it is even more likely to be a 1:1 mapping.

The user agent (by itself) provides you none of those things....

> Why are you pretending that there is some sort of difference between "tracking a user" and "tracking the user's browser"

I'm not? That isn't what I said.

They add extra bits of identifying information to the pool.

You know, https://panopticlick.eff.org/

https://www.browserleaks.com/ is another useful resource to help understand browser fingerprinting and how to combat it.