MD5 still has not been broken. Certain aspect of MD5 have been broken, but as a one-way hash function it remains unbroken. In particular, if I (a non-hostile party) pick a file to hash and publish that hash with or without the original file, a hostile attacker still cannot find the original file (nor any other stream) to generate that same hash.

What has been broken are various scenarios where the attack controls the source file to be hashed. That means that as a digital signature it's no good anymore.

I think it's important to emphasize this because in all the news about weaknesses and attacks, people sometimes get the idea that the sky is falling, and that it's hopeless to really stay secure. That kind of thinking quickly leads in security through obscurity. In fact, the sky is most certainly not falling. Even though many attacks are successful in improving on brute force, even ancient deprecated stuff like MD5 has not been fully broken; indeed it's most critical feature remains essentially untouched (according to wikipedia, preimage attacks have managed to reduce the complexity from 2^128 to 2^123.5 or in other words: nobody cares). And even as a signature, if you had picked MD5 and used MD5 before the attack was developed, you can be reasonably confident that that signature remains valid today.

So yes: don't develop new software using MD5, and retire software relying on MD5 for signatures. And for other applications, if you can easily migrate away from MD5 (and really, sha1 too) - do so! It's easier to reason about security if you don't need to sweat details like this. But if you have an existing application that can't easily be migrated, then don't panic either, just be aware that you need to more carefully consider what it means to have a matching hash (in short it's either the same file, or a different file the original author wanted to appear similar).

MD5 is long-broken as a security mechanism, but as a broader identifier of hashes, a soft match (or matches with potential false positives) might actually be preferred. In this case it might not inherently be a bad idea to use (as opposed to for password hashing).

Actually, precisely for applications such as password hashing MD5 has not been broken. MD5 is broken only in collusion with the author of the original stream to be hashed. For passwords, that would mean an attacker who can pick the password and already has access to the system might be able to pick a password such that other passwords also grant access... who cares?

Of course, irrespective of hash function you need to worry about weak passwords, so you'd need to use salting and an expensive stretching function, or only permit auto-generated passwords. But that's really not a weakness in the hashfunction; it's to mitigate weaknesses in the concept of a user-chosen password.

But why use MD5 anyways? Google has quite an impressive image searching algorithm. Why not just use that? Using MD5 won't do anything because, almost certainly, a false positive is /not/ the same image. You'd need some kind of rolling hash that gets longer the longer the file is. Then do a soft check on that.

Because that would mean they have to train their networks on _huge_ amounts of actual child porn.

> You'd think they would move to a better hash function by now

MD5 is (relatively) fast, and a (relatively) small datapoint to store. You're definitely wanting to store a lot of data points too; it's so so much easier to scan and de-scope OK images.....

The MD5 catalogue's intent is to allow quick identification of suspect material to drop to manual review (little bit of inference there from what I have read about Google, but I'd be shocked if that wasn't true - false positives in this context are not as bad as true positives [of course] but accusing an innocent person of having child porn is damaging).

From the GP:

>That has to be the worst job ever.

It is. I did it for a while working for a computer forensics firm. Not even slightly pleasant. We used to go through millions of images - to be honest, 99.999999999% of it was "normal" (if often slightly wierd) porn.

You do become detached from it after a while.

To be honest; the chats/emails grooming young people were the worst.