Individual www users, unless doing mass scanning or the like, probably do not access 100% of the www's sites (according to Netcraft, most are inactive)
More likely, individuals access only a fraction of the active sites
It's unlikely that individuals access exactly the same list of websites^1
For me personally, the vast majority of sites that I access do not require a UA header
The sample set I use as evidence to support comments I make about this on HN is a list of all sites submitted to HN over a recent period.^2 Currently this is 12,641 sites
1. To have a meaningful discussion about UA header requirements, a sample set of sites must be agreed upon
2. Perhaps HN readers access these sites and possibly access some of the same sites. Of course it's also possible that there are HN commenters who do not actually access HN submissions and instead only read comment threads and submit replies
For example, something like "90% of the current internet is behind Cloudflare, Google, Akamai, etc."
The www is only subset of the internet and "90% of the internet" does not use a CDN
In any event, even if "internet" means "www" there is no evidence to support this statement
https://www.netcraft.com/blog/may-2025-web-server-survey
Further, being "behind Cloudflare, Google, Akamai, etc." does not necessarily mean a User-Agent header is required
For example,
https://www.cloudflare.com does not require a UA header
https://example.com is "behind CF" and does not require a UA header
https://www.google.com does not require a UA header
https://developers.akamai.com does not require a UA header
And so on
Individual www users, unless doing mass scanning or the like, probably do not access 100% of the www's sites (according to Netcraft, most are inactive)
More likely, individuals access only a fraction of the active sites
It's unlikely that individuals access exactly the same list of websites^1
For me personally, the vast majority of sites that I access do not require a UA header
The sample set I use as evidence to support comments I make about this on HN is a list of all sites submitted to HN over a recent period.^2 Currently this is 12,641 sites
1. To have a meaningful discussion about UA header requirements, a sample set of sites must be agreed upon
2. Perhaps HN readers access these sites and possibly access some of the same sites. Of course it's also possible that there are HN commenters who do not actually access HN submissions and instead only read comment threads and submit replies
https://news.ycombinator.com does not require a UA header