... without any technical guarantee or auditability that any of the hashes they're alerting on are actually of child porn.
How much would you bet against law enforcement to abuse their ability to use this, and add hashes to find out who's got anti government memes or police committing murder images on their phones?
And that's just in "there land of the free", how much worst will the abuse of this be in countries who, say, bonesaw journalists to pieces while they are alive?
This will end badly for humanity.
A malware will definitely be created, almost immediately, that will download files that are intentionally made to match CP - either for the purposes of extortion or just watching the world burn.
I'm usually sticking my neck out in defence of more government access to private media than most on HN because of the need to stop CP, but this plan is so naive, and so incredibly irresponsible, that I can't see how anyone with any idea of how easy it would be to manipulate would ever stand behind it.
Even if you do a content aware hash where you break the file into chunks and hash each chunk, you still wouldn’t be able to magically recreate the hash of a CP file without also producing part of the CP.
It's the weights from the middle of a neural network that they're calling a "hash" because it encodes and generates an image it has classified as bad. Experts have trouble rationalizing about what weights mean in a neural network. This is going to end badly.
If this was a hash then it would be as the parent describes, this is at best a very fuzzy match on an image to take into account blurring/flipping/colour shifting.
It's vastly more likely that innocent people will be implicated for fuzzy matches on innocuous photos of their own children in shorts/swimming clothes than it is to catch abusers.
The other thing is, when you have nothing to hide you won't take efforts to hide it - meaning you'll upload all of your (completely normal) photos to iCloud without thinking about it again.
The monsters making these images know what they're doing is wrong, so they'll likely take efforts to scramble or further encrypt the data before uploading.
tldr; it's far likelier that this dragnet will only even apply to innocent people, than it is to catch predators.
All this said, I'm still in support of Apple taking steps in this direction, but it needs far more protections put in place to prevent false positives than this solution allows. A single false accusation by this system, even if retracted later and rectified, would destroy an entire family's lives (and could well cause suicides).
Look what happened in the Post Office case in the UK as an example of how these things can go wrong - scores of people went to prison for years for crimes they didn't commit because of a simple software bug.
The ones that make national news from big busts do, because the ones that don't get caught much sooner and only make local news, because Google and other parties are have automatic CSAM identification online already (server side, not client side, AFAIK), and are sending hits to Homeland Security.
Seriously, folks, we shouldn't celebrate Apple's death grip over their platform. It's dangerous for all of us. The more of you that use it, the more it creates a sort of "anti-herd immunity" towards totalitarian control.
Apple talks "privacy", but jfc they're nothing of the sort. Apple gives zero shits about your privacy. They're staking more ground against Facebook and Google, trying to take their beachheads. You're just a pawn in the game for long term control.
Apple cares just as much for your privacy as they do your "freedom" to run your own (un-taxed) software or repair your devices (for cheaper).
And after Tim Cook is replaced with a new regime, you'll be powerless to stop the further erosion of your liberties. It'll be too late.
Stop. Using. Apple.
But is there a realistically better alternative? Pinephone with a personally audited Linux distro? A jailbroken Android device with a non-stock firmware that you built yourself? A homebuilt RaspberryPi based device? A paper notepad and a film camera and an out of print street map?
Gnu/linux phones have nonexistent security, other than being niche (so security by obscurity at most). And also, they are not yet usable as a daily driver for me personally, at least.
https://puri.sm/posts/defending-against-spyware-like-pegasus...
https://source.puri.sm/Librem5/community-wiki/-/wikis/Freque...
Whether or not I am allowed to check that my entrance has no locks whatsoever doesn’t make it harder to open it. And the reverse, even if I don’t know the details of the lock in my door, it will not let others pass through.
You absolutely can not make that assertion without being able to verify the lock.
For a fact we know that ios has strong sandboxing, secure bootchain and apps are revokably verified.
Not sure about that. No source code. Also, Pegasus.
Can you expand on this point a little bit?
Basically Micay is a legitimate security researcher who created the project and it was later hijacked by the company funding some of it. That company since then try to badmouth Micay at any place they find and is doing shady things on top of the still open source code base. Micay was so professional to destroy the verification key at the time of the forking.
Even if you don't personally audit it, you still benefit from other people doing it. Especially if the software is reproducible (and many packages are).
no idea what the situation is now, but i wouldn't consider a phone that doesn't get those patches (project mainline) on time to be a serious option.
Realistically now we have Tizen and Jolla OS, which had backings from Samsung but nobody gave two damn about it.
I bet even if any of these vanilla mobile OS gets big enough they’ll get bought by the 3 giants and suffocated to death just like how Microsoft sniped Nokia.
Were there any recent examples where they failed in those?
They'll happily do evil shit, and execute it poorly. Samsung are _way_ more likely to leak the unnecessarily and possibly illegally collected personal data they hoover up than Google are.
My hope is that those of you that share my viewpoint will call your legislators and demand regulations or a break up. There are forces of good within the DOJ that are putting together an antitrust case against Apple, and the more of us that lend our voices, the louder and more compelling the argument.
The DOJ is really the last lever we have, and that's pretty good measure for the power Apple wields.
That isn't how I interpret "client-side".
The privacy implications are far more subtle.
It always starts with child porn, and in a few years the offline Notes app will be phoning home if you write speech criticising the government in China.
This technology inevitably leads to the sueveillance, suppression and murder of activists and journalists. It always starts with protecting the kids or terrorism.
Perceptual hashes like what Apple is using are already used in WeChat to detect memes that critique the CCP.
What happens on local end user devices must be off limits. It is unacceptable that Apple is actively implementing machine learning systems that surveil and snitch on local content.
A totalitarian autocracy like China does not need this technology to search for wrongspeech, sadly. You are of course aware that all Chinese iCloud users get their data stored in a special set of datacenters that Apple actually doesn't control.
I wouldn't hold your breath waiting for those regulations to become law in, say, Chine or Turkey or Saudi Arabia. I'd bet even Israel won't pass them, surely NSO have enough political lobbying swing (and probably also suitable blackmail material on sitting politicians).
I wholeheartedly disagree. A world wide goverment would be catastrophic for whistleblowing. Atleast as a whistleblower you can live somewhat safely in a country opposing your own. With a one world government you would have nowhere to run.
And I wouldn't expect them to protect citizen's interests any better than current governments do. Contrary, I think this lack of balance in the world would embolden them further.
Think of GDPR, which is essentially each member of the EU saying in unison, "your product/service must comply with these data protection laws, or you can't legally do business with any of our citizens".
Come to think of it, I wonder if this Apple thing would even fly under GDPR?
Possibly? I’m not a lawyer, but if this is about compliance with a legal obligation, and they’re under that category of pressure? I think GDPR would allow that?
Certainly seems more likely allowed than the stuff Facebook complained Apple was preventing them from doing.
The OP still addresses the inaccurate statement (presented in the form of a question for plausible deniability).
So I want to ask what happens if you have a photo that is falsely identified as one in question and then an automated mechanism flags you and reports you to the FBI without you even knowing. Can they access your phone at that point to investigate? Would they come to your office and ask about it? Would that be enough evidence to request a wiretap or warrant? Would they alert your neighbors? How do you clear your name after that happens?
edits: yes, the hash database is downloaded to the phone and matches are checked on your phone.
Another point is that these photos used to generate the fingerprints are really legal black holes that the public is not allowed to inspect I assume. No one wants to be involved in looking at them, no one wants to be known as someone who looks at them. It could even be legally dangerous requesting to find out what has been put into the image database I assume.
That seems like a realistic attack. Since the hash list is public (has to be for client side scanning), you could likely set your computer to grind out a matching image hash but of some meme which you then distribute.
There have been a number of cases where people have found ways to trick CV programs in to seeing something that no human would ever see. If you were sufficiently malicious I imagine it would be possible to do with this system as well.
State-of-the-art for this field is deep learning, and a /huge/ problem with the DL approach is that you can generate adversarial examples. So for example, a picture of a teacup that is identified by /most/ networks as a dog. It's particularly damning, because it seems like you don't have to do this for particular deep networks, they get tricked the same way, so to speak.
It's a perceptual hash.
https://en.wikipedia.org/wiki/Birthday_attack
http://www.wolframalpha.com/input/?i=2%5E64%20%2F%20%288e9%2...
Now the first few images might be the model completely clothed and not even be porn, maybe there's a picture of her lounging around a pool, then another picture of the pool itself. Still its part of a set of pictures that is known porn.
Heck most porn starts off with actors being clothed (so I hear lol).
No, it'll be done on-device.
> How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif
Nothing. Given that it's only done on their closed-source messaging platform though, nothing is preventing them from reading your messages already.
But yes, it could potentially be used to detect images that the current political party doesn't like.
So... all your hashes will be uploaded to the cloud? How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif,... to profile the users)?
Or will a huge hash database of child porn hashes be downloaded to the phone?
Honestly, i think it's one more abuse of terrorism/child porn to take away privacy of people, and mark all oposing the law as terrorists/pedos.
...also, as in the thread from the original url, making false positives and spreading them around (think 4chan mass e-mailing stuff) might cause a lot of problems too.