Yeah... this kinda seems like an overly sensationalized piece that's just much ado about nothing. From what the article claims vs. what Apple responded with, it sounds like they're interpreting Apple's statement in the most negative manner possible. What's probably actually happening, if Apple is being honest about how similar it is to Google's use of the prefixes, is that the prefixes are sent to Google/Tencent and a bulk list is sent back. Safari checks the list against the actual hash completely locally and blocks whatever matches the list.

So, for example, if someone wanted to visit freehongkong.com with their language set to mainland Chinese, Safari would hash the URL which would give something like "ba7816bf 8f01cfea 414140de 5dae2223 b00361a3 96177a9c b410ff61 f20015ad" and then Google/Tencent would send back every hash and the URL that matches only "ba7816bf" which could be in the order of thousands of URL, or even potentially millions. Safari would then check the URLs that are returned to see which one is the actual matching URL and it blocks the content in the browser without sending any more information to Google/Tencent.

Unless Tencent keeps an active profile and uses machine-learning to create models for every single user of its systems and attempts to tie those models to IP addresses that are assumed to be static, this still maintains complete privacy for the end user. I fail to see how this method is in any way equivalent to what the article is claiming.

I think it's reasonable to be a little concerned. The IP address does not have to be the identifier, it's just another datapoint for the model. For example, how many people in your residential DHCP pool visit Github and Hacker News every day? Whichever IP has those hashes in its pool is probably "you", so eventually a model can be built that watches you move from IP to IP. (Eventually there could be enough data that it doesn't even need to use the "same DHCP pool" as a heuristic, and it can watch you move between home and work or even on your cross-country trips.)

I think it would be difficult to get a strong signal as to what sort of browsing you're doing from this information, but I also think that if life and death ride on what websites you visit, you should turn this off.

Something I should have added to the comment during the editing window:

You also have to be careful about targeted malware, which can give up far more information than these hashes. So I am not really sure where the balance is. It is more concerning when Tencent is running the service instead of Google. A well-targeted ad is better than being disappeared.

Wouldn't tencent, having a list of politically undesirable domains, already know their hashes? Now the browsers that visit them phone tencent to let them know. This is useful when the actual http(s) transfer is under vpn wrapping.

Maybe it won't matter anymore.

They would definitely have a list of the domains but a hash isn't just generated from the domain root. They would have to have a hash generated for every single URL and then they would have to collect every single instance where a hash came up as a "hit" for that hash. They'd then have to model these out to determine which hashes were being hit that were related from a specific IP address which, in my opinion, would be very, very computationally expensive.

A few permutations of each would do it. http://foo.com, with /index.html, /favicon.ico, etc. You don't have to match all of them to mark a user (who in China is 1:1 to IP, IIRC) as visitor of undesirable site.

That doesn't make sense. http://foo.com/index.html would hash to something completely different from http://foo.com/favicon.ico. On top of that, the prefix, which is only the first 32 or 48 bits, would match for thousands of other URLs whose total hash would be completely different. Additionally, hash collisions are a thing so 2 completely different inputs could potentially return the same full hash, much less just the prefix.

You would need a very, very, very large database to keep track of all the permutations necessary to determine a domain name.

I think OP means that the collection of visited prefixes can be indicative. If foo.example.com, bar.example.com, and baz.example.com are "undesirable" sites, and you have requested hash prefixes matching those for foo.example.com/index, foo.example.com/favicon, bar.example.com/index, bar.example.com/favico, baz.example.com/index, and baz.example.com/favicon, you might be able to start making educated guesses that the user is visiting such sites. It really depends how much of an overlap for "acceptable" browsing falling into these buckets there is, or if you can weight certain prefixes as likely to be indicative of visiting "undesirable" sites. You might not able to say this user visited this specific page at this specific time, but you might be able to fingerprint a user as visiting a particular category of sites.

Furthermore, I'm unclear how exactly Safe Browsing is implemented, but if it checks every resource on a page, the set of prefixes for all resources on a given page may leak more data than one query on its own, so you can look for requests for that set of queries in a short time frame to identify a particular page.

While that might be true, this feels like it would be a long way off. The prefixes could potentially match millions of sites and the results are only then matched on the client side. The server would never know which prefixes are the “actual” hit and which are obfuscated prefixes for completely different domains.

Simple: the service provider visits these sites itself, gets hashes of all the resources for the homepage and relevant links (or just spiders them). With the set of hashes, look at prefixes. Users who request one prefix in that set aren't suspicious. Users with many prefixes in that set are.

This form of surveillance would be a game of bingo: for a given "bad" domain you have multiple individual URLs that a browser would hit. Put a few of them on the trigger watchlist, grouped by domain, and when a browser hits some threshold number of the same group within seconds the algorithm shouts "bingo" because chances are this won't be unrelated collisions.

Note also that mass surveillance isn't partucularly concerned about false positives.

The safe browsing hashing algorithm takes that into account. You are supposed to canonicalize and simplify the URL. For example https://example.com/some/path?k=w#hash would have the hash removed, query parameter optionally removed, then path components optionally removed, and finally sub domains removed up to the public suffix plus one, and produce multiple hashes (up to 30 of them).

Or the hashes of all the resources loaded by those pages. That collection of prefixes is going to start getting unique quickly.

Edit: example: P(baduser | hashA )= ~0 P(baduser | hashA and hashB and ... hashF) = 0.75

One prefix doesn't do it, but a bunch together in a smallish time period can fingerprint a site quite well. This is analagous to browser fingerprinting.

But wouldn’t favicon.ico be called from any visit to any of the subpages off the domain’s roots?

It's literally enough that the system can match ~some~ users to just one exact URL. That's a working system, you will see.

If I understand correctly, the full URL is hashed (or at least the first n bits of the full URL). It really isn't computationally expensive to work out undesirable URL's being visited because the undesirable domains are known, and it would be trivial for them to periodically crawl the undesirable domains for all known internal URLs and build a dictionary of hashes.

> this kinda seems like an overly sensationalized piece

From The Register?!?! Surely you’re joking!

Fair point. lol

In my mind, the big deal is that it means a company is actively facilitating governmental mass surveillance. And yes, I believe China does maintain a record for every citizen, and is using machine learning to score them.

https://www.theatlantic.com/international/archive/2018/02/ch...

How do you get that from what's happening here? Apple is actually doing the exact opposite of that. They're obfuscating traffic prior to sending it out to the government entity (Tencent) and only sending partial matches of the obfuscation and then processing the return completely on the local machine. If anything, that appears to be them actively fighting governmental mass surveillance. Even if they do maintain a record of IP addresses, the model for something like that based on URL hash prefixes alone would need to be staggeringly large. They'd probably need more storage and computing power than is available right now.

I'm really curious how you've come to that conclusion and I mean that sincerely. What makes you say that?

They are sending data to what is essentially a state-owned company that can facilitate building user profiles (with enough compute) and is, besides that, explicitly used to censor the web.

What is not to understand?

Apple should send precisely nothing to the CCP if they care about their users privacy on principle.

People unfortunately don't get that and are being slow cooked into accepting being spied on.

I paid for device and then you prevent me from doing perfectly legal action and then notify authorities about it. That's what it is, and coming from privacy hero, Apple.

That’s the entire purpose of this system. It’s useless for creating user profiles.

The purpose of this system is to censor the internet, no? Isn't that enough?

But that being said, I do not believe that hashes are enough, because all they really need is to know you visited one (hash of) a certain website out of thousands. They do not need proof, and they don't need it to work reliably, and they do not need the model to be precise or good. For the system to work in China, this is sufficient. It need not be perfect, it just needs to loom in the background. People will do the rest themselves.

This, I think, one should not forget, especially if you really, really don't see an issue with what Apple is doing.

No it wouldn't. If, as an example, google.com and piracy.com both resolved to the same hash prefix, how would that help you at all? The whole point is that the server sending the blacklist doesn't get the details of the page that's being visited. piracy.com/info, google.com/searchingstuff, and whatthefuck.com/isthis might all have the exact same hash prefix. They don't because I just pulled that out of my rear to use as an example but there's no way for someone to ascertain the content of a website just form the prefixes of these hashes. The browser doesn't even send the details.

Yes it would. All China needs is a correlation. Most users don't input URLs manually, they follow links. So if a user starts from 'taiwan.gov' and goes from there to 'freetibet.org' and 'IhateXi.com'* the firewall would eventually detect from enough matches - even matches with partial hashes - that the user is a 'bad-thinker' and deduct social credit accordingly.

Apple could have tried to avoid helping the Great Firewall (e.g. a proxy could hide times and anonymize IPs) but chose not to.

* fictional names inserted only for example's sake.

I’m not sure I’d call warning people about malware “censoring the internet”. It’s the same system in use in the rest of the world through Google SafeBrowsing. And not just Safari, most other web browsers have some flavour of this as well. But that Google API is not available in China so they use the closest equivalent. AFAIK, the warnings can be bypassed the same in China as in the US, so anger about censorship here seems odd to me.

It’s absolutely not an “overly sensationalized piece” when apple does nothing to offer ip-based controls, such as offering the user a choice of which ips to connect to. You have to install little snitch or get super comfortable with packet fence. Apple certainly has the budget to figure this interface out and has for decades but has not. The issue is not capability but chosen business models.

"Offering the user a choice of which IPs to connect to" is absurd given Apple's target market.

If you would like to disable this particular check, you can do so in settings on both Mac and iOS.

> “Offering the user a choice of which IPs to connect to" is absurd given Apple's target market.

How do you figure? What is Apple’s target market anyway, Mr Confidently Speaking On My Grandmother’s Behalf?

These hashes also reveal domain names. Most users visit many URLs on a small set of domains. If a user requests 1000 hashes that all can map to Reddit, it's very likely that the user is indeed reading Reddit. Another way to look at it: if the same person appears in a crowd on hundreds of photos, it's trivial to notice that there is something special about this person, even though in all cases the person was k-anonymous.

Except that the problem with that is that these hashes likely won't reveal domain names.

For example, the hash for reddit.com/r/gifs would be different from reddit.com/r/funny and so the prefixes would be different for both of them. Unless the requested hashes are saved for every single user, it would be way too computationally expensive for them to get anything useful out of that. Not to mention the fact that hashes would return the same prefix for any thousands of URLs. Narrowing down which domains those URLs are rooted on would be incredibly hard.

> Unless the requested hashes are saved for every single user,

I don't know why or what part you think this is that hard. Do you think a map from User -> set [Requested URL Hashes] is hard to build? Or that building the URL Hash -> set [possible domains] is hard?

Maybe I'm missing a piece of this.

Building something simple to start guessing domains visited seems pretty easy. If a user has 10 URL hashes and the same domains show up in each hashes' possible domains you're probably requesting pages on that domain. If you're lucky and all the pages from a domain fall into a single hash, all it takes is two or 3 hashes from known outbound links to show up to confirm this.

It's not foolproof but hardly infeasible? Or maybe I don't fully understand the algorithm.

You’re ignoring that it’s not the full hash, just the prefix. The prefix could potentially match millions of URLs many of which would be duplicated with random pages from other URLs. You would need a very, very extensive model of every single IP that was mapped to each person and that would not be trivial. That’s exactly why the actual matching is done on the client and not somewhere else.

> Unless the requested hashes are saved for every single user,

It's 4 bytes by request. Google is keeping my whole history, much more than 4 bytes by request, and they do it for advertisers. I have no trouble believing a company partially owned by the government could afford 4 bytes by request.

Let say it's a billion address for each 4 bytes. You do 2 requests, how much of them will be on both list? Let's be generous and say half! You would only have to visit 30 uniques pages on that URL to find the domain. How often do you go on 30 different pages of a website? I feel it's quite regularly.

That's for a billion matches for each 4 bytes prefix, in reality it would be much less than that and there would certainly be much less than 50% matches between each prefix.

You can even do it in reverse even more easily. Go to a bunch of forum that you want to silence the users. Find their URL which allow to post on it. Get the 4 bytes prefix. Now you got a bunch of timestamped comments with username, and they are most likely pretty unique in the Tencent database. Now you found which IP is for which username.

Is it a hash of the prefix of the url? Or a prefix of the hash? Either way, it's deterministic so it's just a more collisiony hash.

On short enough time frames an IP is often a good enough approximation for a person.

No, you are wrong. You don't understand how safebrowsing URL hashes are produced. Read this first: https://developers.google.com/safe-browsing/v4/urls-hashing under the section "Suffix/prefix expressions"

Yes, I do. I linked to that very same site. The prefix is the only thing that’s sent and the hashes are computed as unique for each combination of URL. The docs explicitly say that http://evil.com/foo is unique from http://evil.com/foo?bar.

So any secrets communicated in the URL via e.g. get parameters can be potentially brute-forced based on the hash?

If you have the computing power to brute force sha256 then there are few security precautions a user can take against you.

That opens up forging SSL certificates, signing Debian packages, and a whole slew of other things that you could pull off with more computing power than has existed in the whole of the universe.

Context: https://bitcoin.stackexchange.com/a/41842

Suppose a public PGP key is communicated in the URL.

I don't think there are enough PGP keys in the world that simply trying those keys in a certain prioritized order wouldn't find a lot of keys.

This way you could recognize who someone is talking to based solely on their 'malicious browsing' send hashes.

In general, anytime you know a secret in the URL comes from a low (min-) entropy distribution you can use the truncated SHA-hash as an oracle to (approximately) figure out the secret. Obviously, if the secret is a 128 bit session cookie, you won't find that by brute forcing. Heck, in that case your 32 or 48 bits of hash prefix won't even be enough to uniquely identify the session cookie.

When you get to a 64 bit secret (say again a session cookie), with a 48 bit prefix, things already get dicey. It seems possible that some state actors can already brute force 64 bit secrets, and with a 48 bit prefix, you'd get about 2^16 ~ 64000 (more, but dont wanna compute exactly) candidate session cookies. Those can actually be tried at the website to see if they work.

You're assuming the range of possible parameters is very large. This might not be the case.

>I doubt Apple is actually "handing over IP addresses" - rather it's a feature of connecting directly to Tencent via the IP internet. Though they could conceivably provide some sort of proxying, caching, and batching if they wanted to improve privacy.

I vaguely remember that back when maps.app used google maps, the app would contact apples's servers (presumably a proxy) rather than google's directly.

> Though they could conceivably provide some sort of proxying, caching, and batching if they wanted to improve privacy.

> Similarly Apple isn't "handing over the timestamps" - Tencent can just look at the clock.

If Apple proxied cached and batched as you described, then looking at the clock wouldn't give Tencent a timestamp. (Sure, an upperbound.)

This is why I read the comments on HN. Thanks.