Is it worth all this trouble just to get a Nx space saving?
Because if you just published the 256-bit hashes directly, you could skip the following steps entirely:
> 6. Ask the source of step 1 to give you the full set of hashes corresponding to the prefix from (3) > 7. See if the hash from (2) is in the full hashes you get in (6).
which are the parts that require an actual network call, infrastructure costs, and privacy issues. Speaking of which:
> You can see that at no point does the url being checked go in either direction.
You're not sending the whole URL, but in step (6) you're still telling the source that you're visiting a website belonging to a particular subset of hashes, hashes which the source knows the plaintext of.
I guess the question is exactly how large the set of 'unsafe' urls is. If it's merely in the billions, then (a) truncating the hashes will only get you a roughly ~4x space saving and (b) the 32-bit hashes you send will almost uniquely identify the website you tried to visit. If it's in the trillions, then it makes more sense.
Is there anything else on the entire internet so centralized?
This is quite similar to sending the url. Like there might only be a single site with that hash. Or there might be a handful of which one is far more likely than the others because it has 1000x their dau.
There are obviously ways that you could improve the privacy aspects of this - you could have two versions of the database hashing urls with slightly different initial conditions. A client can keep one up to date and only fetch the other when the get a collision. Assuming SHA256 is pretty good then we can just multiply the false positive probabilities (I think?).
And alternative would be for the client to request multiple regions at a time.
If you want to be a provider of url blocking you do the following:
If you're writing a client that wants to use these providers you do this: You can see that at no point does the url being checked go in either direction.