Companies don't have a monopoly on violence and complete control over you life as governments do, also, they are not unregulated or above the law - this company for example is probably breaking several laws. Spy agencies on the other hand, regularly break laws with impunity or have the laws rewritten to allow unlimited storage (as recently in the UK).
Governments are a much bigger worry when it comes to saving internet history as they have greater capabilities for capture and storage, and a simple policy change in 20 years could make everyone who visited a certain site a criminal liable to deportation or imprisonment.
That's not privacy which is under threat any more. Say you visit a website of random content in January. Website goes out of business, someone else buys the domain and puts flagged content on it in September.
Now... how do you prove you were visiting a different site? Trigger an archive.org call on each and every site I visit? Dig up domain name changes? I've been dealing with DNS for 10+ years and I have no idea how to dig up historical domain owners, especially not with the hide options.
This goes way, way beyond privacy issues.
You're absolutely right though that it's highly problematic if we use automated tools to analyze this kind of data, as many people can end up in the wrong category due to either flawed analysis methods or faulty data.
Again I'd say that currently the greater threat comes from private companies that will use this data to build "shady" rating systems which will affect the daily lives of many people, without giving these people a chance to verify or oppose the data being used against them.
Do you know of any UK caselaw covering situations where people were convicted on the basis of having visited a particular domain and no corroborating evidence was found? Would be interested in reading how that went down.
1. It wouldn't be hard to pollute the logs with noise, and
2. At the ISP level there's no distinction between "browsing a website" and accessing the URL unknowingly.
So let's assume no one will monitor these logs to fight thoughtcrime in real time, and it'll only be analyzed once someone becomes a suspect in a criminal investigation. Would it present as valuable evidence considering the points above?
As to your question: no I don't, but it isn't unheard of for law enforcement around the globe to set up honeypots. But then again there's no need to look into ISP logs because they control the server.
That distinction should be plainly obvious.
Of course it's a completely different if the government forces total surveillance, but the impact on people's privacy can be just as bad or even worse.
The data set contained the complete browsing history of almost 3 million German Internet users, and except for a few popular sites (like Facebook), no URL cleaning/anonymization was performed at all.
So here we have a single provider that is able to capture the browser traffic of 5 % of the population already, and is ready to disseminate this data to anyone who can pay the price (even giving out months of data for free). As there are dozens more companies that collect data using a variety of ways, it wouldn't be surprising to me if you could stitch these individual data sets together to get the traffic of > 50 % of all Internet users, without having any central point of data collection.
So although the government is a real threat to citizens privacy, unregulated private actors are much more dangerous in my opinion.