E.g. 512-bit SHA-2 and SHA-3 may be truncated at 128 or 256 bits if that’s all the entropy you need (and you don’t need to be compatible with the formal SHA2/SHA3-512/256 spec). Here, CloudFlare is truncating to an intentionally low entropy of just 20 bits, not to reduce the security but rather to intentionally increase the collisions. It’s ultimately just a glorified hash table and as any CS student can tell you, the bucket size is just a function of the hash size (v1 of the api: 128-bit hash, v2 of the api: 20-bit hash).
(I don’t like pretension.)
I bet most people who haven't thought about it beforehand or heard the solution, would fail an interview question as "So, we have a database of 100s Million passwords and we want the user to check if his password is inside; without the user actually sending his password.. or his hash; without us outright revealing all our passwords/hashes to the user."
Their solution is a pretty good tradeoff imho.
It's also similar to how we spread mail folders in the past to avoid large directories. Or how Debian splits repo by package prefix: http://ftp.us.debian.org/debian/pool/main/
But then they wouldn’t have been able to give it a fancy name if they called it what it is.
Pretentious: adjective. characterized by assumption of dignity or importance, especially when exaggerated or undeserved: a pretentious, self-important waiter. making an exaggerated outward show; ostentatious. full of pretense or pretension; having no factual basis; false.
If you're suggesting that their analysis is false, you should probably point out why. The self importance part I'm not seeing. Aren't they just stating the facts as they see them?
The reply could very well be interpreted as "You think that Kk-Anonymity is fancy? How primitive of you. This way is the natural way of doing that thing and I would know because I'm the smartest person in the room. Oh by the way I dislike people who are pretentious."
The not-remotely ingenious part was regarding Cloudflare.
If you assume that everyone who is dismissive is actually wrong and just pretending to know stuff, then I guess being dismissive is pretentious.
Pwned Passwords chooses prefixes short enough that any password you wonder about will cause a prefix to be looked up that has lots of Pwned Passwords in it. Was one of them yours? Only you know, this is k-Anonymity.
Safe Browsing chooses prefixes long enough that many sites you look at won't match anything at all. There is still arguably k-Anonymity because the total number of possible URLs is so vast, but that's not their main goal.
I have a suspicion the answers is no due to the way DNS lookup is recursive and so all dns servers would have to implement the protocol. Would there be a way to make this work?
And sure enough:
214943DAAD1D64C102FAEC29DE4AFE9DA3D:2413945
Wow, 2.4 million for "password1". Apparently "123456" is the most common password of all time. Even random phrases and memes I tried were pwned, like "The Lord of the Rings" and "correcthorsebatterystaple"
"The Lord of the Rings" and "correcthorsebatterystaple" match but similar truly random phrases do not, I picked these without looking:
This Duke and Some Circles
impossiblegoosecheesebinder
Sure enough neither of them was in the dataset.
Give it a week...
If you use the API to look up one of you passwords and it turns out to be pretty poor, then the service knows you're highly likely to have been looking up the top (by count) result, so now has the password and your IP address. I appreciate this service provider is well respected, but still, this is also worth noting.
A long hash will identify your password very precisely.
A very short hash e.g. down to 1 byte will have so many matches to be useless.
Cloudflare chose:
> For example; the Hash Prefix 21BD1 contains 475 seemingly unrelated passwords, including:
...and this allows attackers to easily create a list of short hashes of common passwords and try them against matching accounts, as they point out:
> It is important to note that where a user's password is already breached, an API call for a specific range of breached passwords can reduce the search candidates used in a brute-force attack
It's worth noting that haveibeenpwned's API has a really clever design for allowing people to look up their passwords without transmitting them to the site.
It's explained here: https://www.troyhunt.com/ive-just-launched-pwned-passwords-v...
The short version is that you can take the first 5 characters of a SHA-1 hash and hit this endpoint:
https://api.pwnedpasswords.com/range/21BD1
The endpoint returns (right now) a list of 528 full hashes along with counts. You can compare your full calculated SHA-1 hash to that list to see if the password is present in the dump.
The trick here is called k-Anonymity - I think it's a really elegant solution. This technique is written up in more detail here: https://blog.cloudflare.com/validating-leaked-passwords-with...