About that "manual review"... Refer to pages 10 and 11 of the technical paper.

>The server then uses the decryption key to decrypt the inner encryption layer and extract the NeuralHash and visual derivatives for the CSAM matches.

This "visual derivative" term shows up repeatedly. To me, the implication seems to be that Apple doesn't look at the actual suspected image before deciding whether to proceed with a report. Instead, I infer that they only verify whether (as the device reports) the image's neuralhash is indeed present in the NCMEC database. If my understanding is correct, their "manual review" process actually provides no protection at all against collisions or erroneous database entries.

Further supporting this, on page 4:

>Apple reviews each report to confirm there is a match

It only refers to a match, not about whether the image appears to be illegal.

This makes perfect sense from Apple's perspective- who would want to be in the business of reviewing reports of probably-illegal images?- but it means that the references to a manual review safeguard would seem to be false reassurance. Maybe I'm misunderstanding the paper.

Visual derivative is a preprocessed picture itself, and it's part of the "safety voucher".

It may be grayscale and resized/normalized in other ways. Only apple knows exactly what it is.

It's only a matter of time until internet trolls find a way to abuse this. The database is stored on user's devices, so someone downloading it and permuting innocuous images enough until they match the database, and then spreading them for funzies via some underhanded method (wallpaper download sites?)... is not too far fetched.

If they can do a manual review at all, then there isn’t end to end encryption anymore, so I’m missing what the point of client side scanning is.

End to end encryption of messages is by comparison easy as the devices can handle all of that internally. However, losing your iPhone is one of the main reasons to have an iCloud backup. Require a user to come up with a private key and any user who lost it also loses all their data.

Most people don’t really want end to end on consumer backup services, because of the associated risks. If however you don’t want unsecured backups you can handle this manually.

Of course nobody wants the company to actually look at your data, but that’s a separate issue.

The main selling point of Apple is how well integrated the ecosystem is, they could make it super simple to backup the private key on your different devices like watch, tablet and laptop.

Apple doesn't currently have end-to-end encryption for iCloud Photos either: https://support.apple.com/en-us/HT202303

Not today. I think moving CSAM from the server where it's done today to device is in preparation for announcing e2e for iCloud photos.

Apple has gotten a ton of heat over this and they haven't once mentioned that e2e on iCloud is something they're working on or that this technology would make possible, so can people stop spreading this narrative that this is their goal? It's completely baseless.

That the insanity of all of this hand waving. Literally every public photo service does this today.

Most of the time, Apple seems to think through what bad actors will do with their stuff. Their scale and (perceived) reputation basically requires this.

I don't see any reasonable way here. Either somebody looks at the images - i.e. they have some underpaid poorly trained grunt to look at horrible abuse images which is probably illegal to look at whole day, having about 3 seconds to make a decision on each - or what they're doing is just a sham, "Did the computer say 'it matches'? Yes it did! Review complete, match confirmed!" I don't see any non-horrible way of doing non-sham reviews here.

People keep trading away privacy for convenience. We won't have any privacy left if we continue to go down that road. People need to start looking at these tech giants with extreme scepticism. Like we do if companies are from China. We need to think about our American companies the exact same way, and stop trusting them to care about our privacy.

> We won't have any privacy left if we continue to go down that road.

If people look at today's state (phone movement tracking, voice capture, cameras and face ID, etc.) from the mental model of 30 years ago, they would see neither privacy nor freedom. At all.

Sadly, it is not the case of preserving what we still have, if we want freedom and privacy we will have to reacquire them. And a lot of the price will have to be paid in blood.

Before that I thought about dumping Pixel phones for a iPhone. Now I gonna try out CalyxOS on my Pixel 2. Once I finally decided on an upgrade to a Pixel 5 I'll probably give GrapheneOS a shot. Hoping that makes a difference, otherwise it will a dumb phone again.

> who would want to be in the business of reviewing reports of probably-illegal images?

I could think of a couple of companies this could be outsourced to, with strict business and privacy agreements in place, of course, and also conveniently on the lowest end of European minimum wage for the tier 1 reviewers.

> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works.

I've read too many contentious-change memos to believe it's a mistake. These things are authored by committee with Legal always present, and everything is written expecting it to leak. It's just a passive way to manufacture consent by convincing at least some segment of people to doubt their own thought processes. I'd call it "gaslighting" if that word wasn't already massively misused. See also: iPhone 4 owners who were "holding it wrong".

Also, there's a "we can't show you the exact CP image that one of yours matched, because that would also be illegal" catch-22 that basically gives them the power to shroud this whole system in legally-obligated secrecy. Secrecy which will no doubt be abused to hide much more.

I do worry this is basically automating and scaling up the whole "idiot working at the mall photo development booth turns family in for child porn because they took pictures of 3 year old in bath" story that shows up every now and then.

I spent about a year working a retail photo processing lab - I can assure you that what you describe is exceedingly rare.

I saw “questionable” images on a probably weekly basis. Maybe once a month something would come through that I thought warranted bringing my supervisor over to provide a second set of eyes because I didn’t think it warranted calling the police over but wasn’t entirely sure. An example of that that comes to mind was a series of photos that appeared to show a young woman bound and gagged to a pipe in a basement area - but she was obviously of age, her eyes in the photos didn’t look fearful, and the rest of the roll showed her free and apparently happy. Finally, I was the person who accepted the roll of film and knew that she was the one who dropped it off. We didn’t call anyone on that, but I figured it wouldn’t hurt to get a second opinion. When she came to pick them up, I did ask her to review them and make sure everything was OK with them.

Twice in about a year we have photos come through that were obviously what’s now called “CSAM”. In both cases I called the police without consulting management then told them afterward. Also in both cases, the negatives were put into the store’s safe and the owners were arrested on-site when they came to pick them up.

All of this is to say - photo lab techs see some shit. If they called the police every time there was a picture of a kid in the bath, police brutality would be at an all time low because they’d not have time to respond to anything else. :)

reading through Daring Fireball's technical discussion of it https://daringfireball.net/2021/08/apple_child_safety_initia... I suppose does much to allay my worries though, although my natural distrust of authority and the powerful still makes me not like anything like this - no matter the spin.

It's exactly that though. Except instead of the "idiot" it's "completely dumb neural network".

Sometimes the slippery slope is not a fallacy, but the natural consequence of seeking more control. This feature can be horribly misused in the future, for example, to find people who have sent/received/downloaded Tiannanmen pictures, because instant messaging pictures can be automatically added to your albums.

You can't trust a feature whose only impediment from being abused in the future is such pinky promise.

Add hashes of police uniforms and now you cannot take pictures of law enforcement.

Add hashes of politicians. Businessmen in suits. Army uniforms.

At the end you have a weapon that can only shoot civilians.

Ok but PhotoDNA has been doing this scanning for a decade, I believe, and none of these things have happened despite how easy it would seem to be?

Or have Microsoft, Google, Apple, Facebook, etc., been doing this with such stealth that no-one has actually noticed it happening yet?

Those aren't run on edge devices. Apple's model saves them money and creates a capability - local scanning - which didn't exist before.

Recall that apple devices include an ML accelerator, at some point or another, the next step down this slope will be adding this to the display pipeline.

PhotoDNA and similar are done in on-premises machines, nothing on your own private phone.

Do you know how discovery works in court? I'm guessing not.

Once a hash matches, law enforcement would need a subpoena to access the raw image. If a person were to be arrested, that evidence would be turned over to the defense.

It would be very obvious that a picture of a police officer was not child pornography.

Are you under the impression people are jailed for hash collisions? Because that's not the case at all....

I don't think the suggestion is that a hash of a police uniform will be used to convict of CP but rather the idea that a new law will be passed that - in order to protect the police from 'harassment' - will outlaw taking pictures of the police.

So in that case in discovery the picture of a police officer would be evidence that you broke that hypothetical law. The technology deployed to protect the children opens up the possibility of its deployment for other things later, based on legal requirements of course.

For example don't take pictures of copyrighted material would be something people might want to work on.

> would be evidence that you broke that hypothetical law

Hypothetical situations are not an argument to real world discussions. Sure, if that happened, it'd be terrible. But, that's not happening....

I head over to my friend Midev's house with precursor chemicals for VX. You complain that having the precursors to VX lying around is not a good idea. I point out that creating VX is purely hypothetical. Sure, if that happened, it'd be terrible. But, that's not happening.

Apple is deploying a technology with few legitimate uses that makes terrible things not only possible but easy all without the voluntary consent of the device's owner.

Yes, they are. The point is that this change make this hypothetical too close to being true so the change should be abandoned. It skews the power balance too much into the hands of a small number of unknown people so the rest are protesting.

It's not hypothetical https://reason.com/2021/07/27/florida-bill-criminalize-filmi...

Police: "Did he murder you yet? No?! Stop wasting our time with hypothetical situations and come back when you've been murdered for real!"

I think asking ourselves if this is a slippery slope is a valid part of discussing what's actually happening right now.

Appealing to the slippery slope is itself a fallacy.

"While this image may be insightful for understanding the character of the fallacy, it represents a misunderstanding of the nature of the causal relations between events. Every causal claim requires a separate argument. Hence, any "slipping" to be found is only in the clumsy thinking of the arguer, who has failed to provide sufficient evidence that one causally explained event can serve as an explanation for another event or for a series of events"

You need to argue things that are actually happening. Appealing to hypotheticals, especially when technically incorrect, serves no one.

What's actually happening is that a mechanism is being introduced that makes it easy to censor anything Apple wants censored. Apple promises to only use it for child pornography.

Apple, however, is still not a sovereign state, and as such must bow to the wishes of actual sovereign states. Sovereign states have proven again and again that they will grab as much power as they can, and doing it insidiously, in the form of a private database of hashes of undesired content, is especially attractive to them.

This is not a hypothetical. For a real-world example look to England where its nation-wide internet blocking system is already used beyond its original scope. Or think of what countries like China will certainly do with such a mechanism. Scope creep, in the form of power grabs by nation states, is a realistic concern, based on vast historical experience, not a fallacy.

Apart from the concern of scope creep, there is also the concern of false positives. When deployed on such a scale, there will undoubtedly be perfectly legimate images being flagged. I'm not happy about my phone containing software that's always vigilant, ready to ruin my life over a false positive.

Arguing that someones point is false, simply because it contains a fallacy, is also a fallacy: https://rationalwiki.org/wiki/Fallacy_fallacy

>You need to argue things that are actually happening.

No. You don't get to control the narrative by saying that. You have to discuss things that could happen when discussing things this invasive.

ok first off that slippery slope referred to here is the domino theory - which is a faulty idea of the causality of events - we can say a slippery slope regarding events is likely faulty because unless the causal chain between any two events is very tightly linked our understanding of causality is not sufficient to understand it. That is to say we have a very tight link between a domino falling, hitting another domino and toppling it, but we do not have a very tight link between a country going communist and then another one later. Hence, every causal claim requires a separate argument.

The slippery slopes being discussed here are not about events and the causes that link them, but about the applications of technology and to a certain extent about laws being used beyond their original mandate. This kind of slippery slope is a logical one.

Thus the technology being used for this mean that it can be used for other things - what kinds of things can it be used for? Are there things that people would like to make illegal or that are illegal now that this technology can be used catch people infringing on these hypothetical laws. Thus - if we allow this technology in our devices now are we opening ourselves up for other potential uses for the technology in the future that will hurt us.

If this argument seems the same to you as the domino theory and that we must take every hypothetical problem when it actually occurs I wonder how you are ever able to plan for any eventuality?

Does this mean you believe we should throw away all conspiracy[1] laws?

[1] https://en.m.wikipedia.org/wiki/Conspiracy_(criminal)

I understand your argument, that a court (in a democratic country with a rule of law) would expect that the investigators extract the original image from the accused's phone with the appropriate chain of custody of the evidence.

However, that does not mean that while this investigation is under way, the accused is now go about their business.

For CP, or other crimes of violence, that makes sense.

But if laws are passed against peaceful protest, then that means someone could be held in detention while those photos are investigated. An anonymous photo now is trackable.

> Once a hash matches, law enforcement would need a subpoena to access the raw image.

Not if Apple provides it to them without a subpoena.

That still doesn't change how discovery or the courts work.

Depends on the court: https://en.m.wikipedia.org/wiki/United_States_Foreign_Intell...

It's like you think everyone operates under US law, or don't give a shit about anyone who doesn't.

Have you heard of the chilling effect?

Yeah, which is irrelevant to the discussion. This thread is about photos of law enforcement and politicians used to convict people on CP charges.

Let's stick to the topic of discovery and how courts work.

Let’s conveniently ignore all the facts that speak against the planned dystopia, and focus on the positives of having all facets of your life constrained and controlled by authorities.

I'm happy to have that discussion, I think it's critical and we need to update laws to protect privacy.

You still don't get to be technically incorrect, though. You still need to know how courts work, if you want to participate in that discussion.

You have a lot of trust in your courts. Let’s hope it is not misplaced.

Not at all! Courts are a joke.

You still need to know the process, though. Images of police officers are not being used to convict CP charges.

That was not the claim. It seems you are deliberately/conveniently remaining naive at this point.

If I learned anything in my lifetime, it’s that “slippery slope” is not, in fact, a fallacy at all. Rather, it’s a rule of thumb.

There are plenty of examples where it is fallacious though, often absurdly so. The important distinction is: does A abstractly "lead to" B, or does A directly enable B, but for some will to use it that way.

For one example, it was always absurd to suggest that gay marriage would lead to legalized pedophilia, bestiality, or marriage to objects.

Meanwhile, trolls and staunch intellectuals demanding "enough" evidence think they can gatekeep what's acceptable to notice is already slipping and gaslight others for noticing and worrying, only to find that years later, the Overton window shifted and newer generations were none the wiser, bringing the things that were previously unacceptable into the mainstream with reckless abandon.

(edited for brevity, expanding more on this in a reply...)

I'm going to get a little facetious here:

The corporation that is the U.S government acts just like one: roadmaps and planning ahead for radical policy changes to occur within longer spans of time to signify progress (or something), and then absolutely losing their shit if an opposing candidate wins & gets in the way of their progress, as we observed these past 5 years.

Acceptance of pedophilia has been set in motion for several years now. One only need look at some of California's SB-145, the ever-expanding reach of public schools teaching sex-ed to kindergartners, and the N number of drag queens sexualizing themselves in front of young children in public libraries and schools with grand applause and media gushing as these facilities move a few steps away from becoming brothels.

(See what I did there? Surely our libraries won't become brothels, that would be absurd! I mean, the chances of that happening are 2nd to none; and if 10 years from now, libraries still aren't brothels, then I'd have been wrong after all, and my playful exaggeration will have expired. A gatekeeper would have done their noble internet duty to inform me just how idiotic my suggestion was from the start, to be sure. The joke's on them for losing 15 minutes crafting the perfect response to my alarmism that will definitely change my mind.)

We had none of this just 3 years ago, but I'm pretty confident I'll be gaslit in response to my highly misinformed and "hateful" comments, as it goes. ;) That's no matter though. I take responsibility for that by standing by what I say, not bending to the winds of outrage.

The only example you mentioned that is attributable to government is sex-ed.

Sex education isn't about teaching you how to have sex, it's "here's all the reasons you need to be very careful with sex".

In my classes I learned about many different STI's and the dangers of unprotected sex. Not once was I taught a Kamasutra position or what to do with my fingers.

For younger kids I assume the curriculum would be more about what kinds of behaviors they need to be careful of and immediately warn other adults about.

I suppose the name is very unfortunate because a lot of people seem to think sex-ed is about getting young people to start having sex, when in fact it has the opposite result and we can see it in statistics.

Instead it lead to things like being sent to jail for calling his daughter she: https://nypost.com/2021/03/18/man-arrested-for-discussing-ch...

You're implying a slippery slope fallaciously. Legal protections for transgender people does not directly follow from legalized gay marriage.

Also, that father violated a gag order, and acted against the decisions of the both custodial parent and his child's medical professionals. You're being disingenuous to misrepresent that as "for calling his daughter[sic] she".

It is only a fallacy when it’s not backed up by solid reasoning. There are obviously slippery slopes. And sometimes they aren’t there, even though people claim they are. It’s a shame that some have been conditioned to immediately make the connection between the two.

It’s almost like that conditioning benefits someone.

And if the landscape is changing then the sliding has already started.

Just because the slippery slope argument is a logical fallacy doesn’t mean we can’t use our minds to foresee possible next steps.

I'm surprised to hear this falsehood being repeated here. There is nothing inherently fallacious about slippery slope arguments. Like almost all rhetorical tactics they can be used fallaciously. The fact that slippery slope is not inherently fallacious is very well documented in the academic literature on rhetoric.

https://en.wikipedia.org/wiki/Slippery_slope

http://www.csun.edu/~dgw61315/fallacies.html

The slippery slope argument isn't even really a logical fallacy. In fact, I can think of a number of circumstances where it turned out to be true.

edit: here's an example of the slippery slope "fallacy" in action

"First they came for the socialists, and I did not speak out— Because I was not a socialist."

It is a logical fallacy in that the consequences do not logically follow from the proposition. However that doesn’t say anything about whether they are likely to follow in practice, only that there is no logical proof that they follow.

In this case, the arguments against this mechanism are generatly slippery slope arguments - logic doesn’t guarantee the bad outcomes. However it’s quite reasonable to be concerned that the bad outcomes will happen because of the actors involved.

>isn't even really a logical fallacy. In fact, I can think of a number of circumstances where it turned out to be true.

This represents a very common misunderstanding of fallacies, IIRC it's called the Fallacy Fallacy: fallacies don't give you any information about the truth of the conclusion, they only indicate that the logic that ties that conclusion to the premises is unsound. So "slippery slope fallacy" means "worse things don't logical follow in increments from better things" but that doesn't mean that in any real situation "worse things follow in increments from better things" is not true.

The most common example of this misunderstanding is probably Occam's Razor, it says nothing about whether things are or aren't more complex in any real situation, only that it's easy to reason about things if you don't add additional aspects that aren't needed.

Not only is it 'not really' a logical fallacy, it really is not a logical fallacy. Check my post on the parent for references.

Pictures is just a pretense. Step #2 will be URLs of pictures that users send each other using the phone's keyboard. Step #3 will quietly expand the definition of a URL to any keyboard input. At that point, Apple will have a god's eye view into all communications of their users, be it over WhatsApp, Signal, SMS or an online forum. Imagine having a dashboard that shows in realtime with gps coordinates how many text messages contain the word "enough"! The same scanner module can detect certain words in phone calls. What dictator wouldn't want that?

If there ever is a showcase of the slippery slope fallacy this comment would be on it.

Step back for a second and think about why perhaps falling for the ‘but the children’ trap tells you Apple wants to go in the absurd direction you are pointing. Think about if it makes any sense at all they would want to go that way, think about why they would even let you know and think about how plausible it is for these steps to lead to each other.

It's not fallacy. There's huge demand for such monitoring and few countries are as competent as China to do it themselves. Apple already works closely with CCP, so it won't face any moral dilemmas. The only obstacle is damage to reputation in the US and Apple is assessing the scope of this damage right now.

If Apple wanted to do all that they can, they don’t really need a special feature for that and tell te world.

Apple cares about reputation and that's why they test the waters first with these announcements.

Do you really think that "Apple is capable of doing this" is a meaningful rebuttal to "Apple will actually do this"?

I started writing an explanation of how you're mistaken, and then I realised just how quickly you went from Apple using a very common method for identifying CSAM (which Facebook already uses, yet I've not really heard much in the way of complaints about that) to Apple wanting to listen for keywords in phone calls, and just how unlikely it is you'd reasonably think through my response.

> which Facebook already uses

A company looking at content uploaded to servers they own is quite different from a company spying on the contents of a device they don’t own.

Perhaps you're the one who's wrong though? Your response makes no sense other than to grandstand how hasty and unreasonable the OP is being. You could have just omitted it and we'd be no worse off.

Does Facebook scan local files?

A good way to think about any potential law like this is to imagine your worst enemy or most hated politician was given power to abuse said law, and then think about all the ways they could use it to make your life miserable

always think about the worst case scenario when it comes to privacy, because that will be the end result

To me we don't even consider the actual worse case scenario.

Is there any doubt 50 years from now all this will be on algorithmic autopilot for digital authoritarianism? All societies will have Chinese style monitoring but to an unimaginable degree even to people in China right now.

People outside IT were barely even using the internet 25 years ago. We are at the very start of this and already too far gone. You just have to enjoy these times and what we still have.

It's not a mistake, it's a rhetorical trick - to present somebody opposing your actions as a misinformed idiot that just doesn't know what they're talking about, and once proper technobabble is deployed every reasonable person would realize their mistake. And if they don't, they must be much bigger idiot than we thought, there's no other reasonable conclusion.

And HN's mistake is thinking any of this will matter. Apple only cares about money, always has, always will. And I'm willing to bet they are going to continue beating their revenue and profit record quarter after quarter. When their actions do not affect the one thing they care about, why should they care what anyone says?

Ah ok, so as long as it is profitable ethics don't matter. Got it.

Yes, unfortunately that seems to be the world we live in.

The parent poster is not advocating for profits over ethics - just pointing out the reality for companies like Apple etc

Companies don't have to be ethical, they just have to be legal.

This is the sort of reasoning that got us into all this crap in the first place, and it certainly won't get us out of it.

Not saying its good but these are the facts. Companies try to maximise profit within the boundaries of the law.

> This is the sort of reasoning that got us into all this crap in the first place

I don’t consider us in “crap”. I welcome Apple’s decision.

So just like everyone else?

We hold people to certain standards and can do the same for companies.

I don't agree with Apple's move here either just explaining that they are not doing anything illegal no matter how much we may not like it. Of course we can choose not to buy their products but do you think the average person considers it before they make a decision on their next phone?

Precisely. I just have a problem with them being hypocrite.

> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works.

They don't believe this. The lack of understanding trick is often used by governments and corporations because it robs the opposite party the choice to oppose, as opposition is framed as lack of understanding instead of a political choice. It's a semi-elaborate way of calling idiots people who disagree. This is why governments "educate" on topics they don't people to oppose.

They make the premise so innocuous that it would seem unreasonable to object, eg who would object to keeping kids safe?

However it's what comes after this, the slippery slope of what to enforce and not. The genie doesn't go back in the bottle.

Exactly. It's the same argument when Corona contact-tracing was discussed and later implemented.

First using paper forms and later using apps.

Why would you object to limiting the spread of the Coronavirus ?

Fast forward to the present where law enforcement in Germany, Australia and Singapore have used contact-tracing for other purposes.

So no to this plan from Apple, they must discard all plans of client-side scanning and instead increase their privacy stance.

Could you please expand on the other uses of contact tracing in those countries?

This is a compilation of what others have posted:

- Germany

https://www.golem.de/news/hamburg-polizei-nutzt-corona-konta...

https://www.ccc.de/de/updates/2021/luca-app-ccc-fordert-bund...

- Australia

Australia, WA, check-in data: https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urg...

Australia, WA, border pass data: https://www.abc.net.au/news/2021-06-17/g2g-app-data-accessed...

Australia, VIC, check-in data: https://www.theage.com.au/politics/victoria/police-sought-ac...

Australia, QLD, check-in data: https://www.abc.net.au/news/2021-06-29/queensland-coronaviru...

- Singapore

https://fortune.com/2021/02/01/singapore-covid-data-tracetog...

https://www.straitstimes.com/singapore/politics/police-can-a...

https://www.straitstimes.com/singapore/proposed-restrictions...

https://www.straitstimes.com/singapore/politics/bill-limitin...

Thank you, that was very thorough and enlightening.

> Fast forward to the present where law enforcement in Germany, Australia and Singapore have used contact-tracing for other purposes.

I don't understand how people were so naive to install those applications. I did the exact opposite: whipped my phone and bought two cheap ones so I can separate my calling phone (which I leave home), from my media phone (no connections), and my GPS phone (no connections aside from GPS and turned off most of the time).

Yeah, this is my reading of the situation too. From a technical point of view, what Apple is doing seems like a reasonable approach to me; what I absolutely do not trust is any claim that it will never ever be expanded to include, say, governments requiring similar tech be applied to all messages before they are encrypted in order to look for dissidents… and then generally at opposition parties. (And of course there are enough competent technologists available for making custom encrypted chat apps that the hostile governments could only achieve their goals by putting the AI in the frame buffer, which opens the door to subliminal attacks against political enemies — targeting the AI with single frames such that the device user doesn’t even notice anything).

But I know I have paradoxical values here:

On the one hand, I regard abusive porn as evil. I want it stopped.

On the other, I have witnessed formerly-legal acts become illegal, and grew up in a place where there are sexual acts you can legally perform yet not legally possess images of. I think the governments (and people in general) follow a sense of disgust-minimisation, even when this is at the expense of harm-minimisation.

And any AI which can genuinely detect a category of image can also be used to generate novel examples that are also in that category without directly harming a real human.

I don’t have a strong grasp on what I expect the future to look like. My idea of what “the singularity” is, is that rather than a single point in the future where all tech gets invented at once, it’s an event horizon beyond which we can no longer make reasonably accurate predictions. I think this horizon is currently 2025-2035, and that tech is changing the landscape of what morality looks like so fast that I can’t see past that.

> A cryptographic hash + file size combo is unlikely to have a false positive within our lifetime

The chance of a "technical" false positive is tiny, but they need to look all false positives: If some joker sends you a lewd picture through WhatsApp, WhatsApp by default saves every picture to your photo library, then you are now on the naughty list.

Good luck trying to explain yourself out of that one.

The context there was that a typical CSAM hash like PhotoDNA is NOT a cryptographic hash, but a perceptual one, which DOES have notable false positive issues.

WhatsApp saves all received images to your library? That seems hard to believe. I can't imagine that anyone would appreciate having their personal photo collection cluttered with other peoples photos and all of the meme images that people share.

It's a setting, called "Save to camera roll". Default is on.

Strange. Is that a desirable feature for WhatsApp users? I must be overlooking a use case that makes that a sensible default.

>these features are part of an “important mission” for keeping children safe.

The other mistake is to think that removing privacy with longing effects on society [1] and thus attacking the very fabric of a free democratic society respecting human rights would not make children any safer. When they grow up they possibly would have to live in concentration camps or Chine-like regime. It is very safe, but not sure it is very pleasant.

It's about values. It should be absolutely illegal to install any spyware on personal device without warrant

[1] https://news.ycombinator.com/item?id=28084578

[2] https://news.ycombinator.com/item?id=28096948

You nailed it. Apple is so flexible with their morals when money is on the line you know this will be abused by some authoritarian government. They will ask apple to scan for some content, apple will say no, the government will ban apple from selling their products, and suddenly apple will change their mind. Next thing you know a huge swath of journalists or dissidents is being disappeared to reeducation camps. Apple is working overtime to wrestle the “most evil company in the world” title from google and amazon.

yes. i live in a small 8 million strong "valley" where the government is hellbent on destroying people. https://kashmirobserver.net/2021/07/10/11-govt-employees-inc...

they sacked these government employees because they allegedly were engaged in "anti-national activities", no courts, no trial, no hearing. just sentencing that since you are allgedly found to be engaged in such activities, your employment is terminated and there is no recourse. https://www.organiser.org/Encyc/2021/8/3/Govt-s-policy-of-No...

here they have decided to not give passports to protesters or give then government jobs because "anti national activities" again the same trope. this might be a small issue but here is a jounalist https://thewire.in/rights/kashmiri-journalist-masrat-zahra-p... whose father was beaten to a pulp because she being a decorated journalist had criticized the government, they slapped terrorism charges on her and german government gave her asylum so they did the next best thing, hurt her parents because why not. now if she didnt have her passport because she was already a known suspect of "anti national activities", asylum might be a little difficult so they want that.

i know 100% indian government and for that matter pakistan government as well will use this technology for "national interests" in finding out protesters and dissidents and they either do not even go for trials or if they do, the same are sham ones so doesn't matter if they make a show of a trial because the guilty are condemned already.

I see whataboutism (google drive scans! ms scans!) and "here are the technical details of why we can't just look..." . The bottom line is that none of that matters. the 99.999% problem is Apple is spying on you on your. own. device. I don't know how many techies I've had to shoosh and point that out. It doesnt matter what kind of hash or algo or "layers of protection" are being used. What they are doing is inherently wrong and totally the antithesis of their privacy mantra. It will also inevitably be used by governments for ant-humanitarian purposes to imprison people for thought crimes against the state. Unless they (Apple) backpedal this abomination soon I'm definitely minimizing my apple purchases in the near future and moving off the platform entirely long term. Even google isn't so bold as to put their spyware on their devices to police you for the government. Suddenly ads don't seem that bad ...

The mistake here is thinking iphone is your own device. If they can view anything stored there, remove anything from there, control anything you install and run, disable it at any moment they wish - do you actually own it, or you just granted the use of it as long as Apple approves of it? Apple certainly doesn't ask your permission to do anything on "your own device" - but you have to ask theirs. Is that how ownership works?

Having ditched Apple laptops this year (after 18 years of Mac computers) for Linux (on System76 hw), I've been wondering what it would take to pry the iPhone out of my hands. I think I've found it.

No, Apple fully understands the situation, and in fact, you don't see many Apple fanboys in the comments defending the firm for once because even they know it's wrong.

Apple just communicates this message because that's what they need to say for their business, that's all.

And they can do so because they also know this uproar will disapear like tears in the rain in a short time if they keep smiling. Their bottom line will not be affected because people don't care.

Remember that now, Bill Gates, who we used to associate ti Satan in the 90s, is now viewed as a hero.

Remember that about 1 american out of 160 is working at Amazon, a company with the reputation of treating their employees terribly. Most people hence knows somebody impacted by this, but don't stop shopping at amazon.

Remember you could say you grab them by the pussy and be elected president.

Remember people gives in mass their data to someone who said "they trust me, the dumb fucks".

Remember that when somebody risked his life to unveil a massive conspiracy to spy on the entire world, the person has been hunted as a traitor.

Remember that one president can lie about WMD, go to war against the vote of the UN, kill thousands of civiliands, spend 600 billions of dollars there while poverty is rising at home, and suffer no consequence whatsoever. Most people have forgotten already. Hell, my girlfriend don't even remember the name of the guy.

But it's you that have to be monitored all the time for CP. And you will be punished harshly if you behave badly, not just according to law, but also according to social contracts, both which powerful people can escape now with money and PR.

That's the world we are living right now.

It's just a matter of time until someone can create a "ThisChildPornDoesNotExist" site, then send links to targets to have their lives ruined. I am extremely worried by humans policing the Intertubes, and even more worried by algorithms doing so because they make the perfect excuse to do sloppy checking. See Youtube DMCA related takedowns for multiple examples.

> It's just a matter of time until someone can create a "ThisChildPornDoesNotExist" site, then send links to targets to have their lives ruined.

“Their” in this case referring to the sender, which makes the particular problem somewhat self-limiting, right?

Probably so, but a more smart malware that grabs fake images from an encrypted server somewhere, then plants them in the victims PC/phone/whatever, then deletes itself, isn't rocket science.

This still means iOS 15 is adversarial and *Apple lacks credible neutrality*.

> Apple's mistake is that they seemingly believe there is pushback because people misunderstand how it works. The reality is more nuanced: People understand exactly how it works, and how it works is that it is turn-key onboard spyware, that Apple pinky-swears isn't being used wrong today.

It's not a mistake. It's gaslighting.

Isn't perceptual hashing necessary for this application to prevent simple modifications (e.g. resizing, filters) from changing the signature?

Apple: our staff can look at your tits if you take a sexy picture and look young enough or some other magic button.

Hm. Seems like a good attack would be take a celebrity, photoshop some nakedness, and put that in the database. Output: real photo of said celebrity.

This is not acceptable.

Seriously, the backdoor is in place.

Now all they have to do is keep adding hashes to the list and voila. Trump supporters = terrorists = you're in jail because of political games.

Quote:

> and more than a few are worried about the implications

>"A cryptographic hash + file size combo"

SHA already uses the length in the output hash. Having the explicit length is quite superfluous

> Having the explicit length is quite superfluous

Not entirely. It makes it so that, to achieve a "full" collision, you have to ensure that the sets of data collide both in SHA hash and in length, helping to prevent attacks that rely on appending/prepending/removing data (for example, "length extension attacks" involve manipulation of the hash by appending data).

TL;DR: It is harder to find a collision SHA(B) for SHA(A) if you add the additional constraint that the length of B must match the length of A.

This is completely wrong.

The known collision attacks for the MD-family and SHA-1 all in fact produce collisions with the exact same length. The method used necessarily does this.

> This is completely wrong.

Which part? The fact that storing "length" along with a hash is not superfluous?

You can probably find many things which have a SHA hash of "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb" (infinite things, if we assume arbitrary-sized inputs), but you can only find ONE thing which has that hash and has length 1. I just made it impossible (not just unlikely) for you to find a collision.

> The known collision attacks for the MD-family and SHA-1 all in fact produce collisions with the exact same length.

Emphasis mine. And note that I did not claim otherwise in my comment.

> Which part? The fact that storing "length" along with a hash is not superfluous?

The part where you make a false claim out of ignorance.

> You can probably find many things which have a SHA hash of "ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb"

No reason I should go looking for such things. You're the one making the false claims, if you have found "many things" with that hash then list them to prove your point, otherwise go away.

> The part where you make a false claim out of ignorance.

Which false claim did I make? I'm still waiting...

> No reason I should go looking for such things. You're the one making the false claims, if you have found "many things" with that hash then list them to prove your point, otherwise go away.

You don't need to look for those things. By definition, you know they exist. I don't need to find or enumerate all primes to know that an infinite number of them exist.

For more information, see here: https://en.wikipedia.org/wiki/Pigeonhole_principle

By definition, assuming arbitrarily-sized inputs, there are infinite messages that collide to the same hash value.

But, don't worry... it is clear you have no actual meaningful point to add, so I won't continue this conversation with you any further. Have a nice day.

You are misrepresenting or more likely have simply misunderstood the Pigeonhole Principle. Which I guess makes sense for somebody who didn't understand why length extension matters. It does not prove that any particular output will recur, and what you've got here is one very particular output.

Again, you need actual examples. Not handwaving, not the unwavering yet entirely unjustified certainty that you're correct, you need examples. And you don't have any.

Again, which false claim have I made? Be specific and quote me: you need actual examples, not handwaving.

Until you do that, I'm not pursuing this conversation any further. Have a nice day.

EDIT: Also, if you do want to have a conversation, make sure to stick to HN rules and talk about what is being discussed, rather than about me. Thanks.

Yeah, having to tack anything onto a secure cryptographic hash for practical collision avoidance means you should just use a longer hash.

I disagree. Google announced the same thing for Google Drive and didn’t receive this level of pushback. If you’re Apple, the only way that’s possible is if you explained it badly (i.e. differently than Google did).

I’m guessing their real prerogative is to keep unlawful content from ever reaching their servers. You wouldn’t know that from reading their press releases though.

Why wouldn't they show a pop-up on the phone highlighting which images may be problematic and that report is being sent to the government? I might change my mind on uploading these things to my iCloud. Can't wait for all examples of false positives that will surface.

Why wouldn't they delete them directly, but rather send report to the government?

Why wouldn't they go even further and if image detected on the webpage that is problematic => It doesn't allow it to get to my filesystem either, so that I don't get into trouble if for some reason police decides to search my laptop.

All relevant questions. Answer is the only one: so that when police knocks on your door you were blissfully unaware.

Quite honestly this is all too sad. Again this argument about CP that everyone will buy, but which doesn't work in the wild. There are some bad people out there, but if these people are into buying or creating CP => they will rarely upload it into iCloud. Especially now, when everyone knows Apple is looking for it. So they will go even deeper into hiding, while everyone else will have to live with devices that are spying on them...

The difference is that Google doesn’t advertise itself as a bastion of privacy. Google spying on you is expected. More consumers also pay for iCloud vs Google Drive. Expectations change when you’re paying for a service.

Isn't the difference that with google drive, they scan the files you upload to their servers? And, this latest apple debacle is on scanning files locally on the device itself?

If Apple is to be believed (and doesn't change anything later) then they're doing the same thing. They're scanning the things that are being uploaded to iCloud. They're just doing it locally instead of doing it after they're on the iCloud servers.

> people misunderstand how it works

You can literally read any thread on HN or Reddit to see this is the case. Thousands of comments about facial recognition and AI, completely misunderstanding how it works. The EFF article itself was FUD.

The EFF article starts right off claiming a backdoor, which is completely incorrect. Taking hashes of client-side content is not a backdoor, and the EFF should be embarrassed for not understanding that.

This client side scanning of local files can be easily extended to scan the whole device. That's why the eff was calling it a backdoor. All it takes is for the right govt to put the right preassure on apple. In a way it's kinda worse than a backdoor since your phone is actively reporting any suspicious looking files to the authority all the time on its own. The govt dosent need to actively look for it. Just add the appropriate has of what it wants discovered to the db and wait

> This client side scanning of local files can be easily extended to scan the whole device

Correct, Apple can access unencrypted files stored on your device.

> That's why the eff was calling it a backdoor

Accessing unencrypted files on a device is not a backdoor

> actively reporting any suspicious looking files to the authority all the time on its own

Yep, that's a problem. Not a backdoor, though.

What is your deal. You are out here arguing in bad faith all over this thread. What is your motivation?

What's the bad faith? What's single thing have I said that is incorrect? It's not breaking E2E encryption. No one is locking people up for political photos. It's not using facial recongition.

I don't like FUD driving discussions. I think we need to have serious discussions about this technology. I think people that don't understand it shouldn't participate....

> It's not breaking E2E encryption.

It's not breaking it, but it is rendering it moot. Why even bother with E2E encryption if you can't trust your endpoint? Would upgrading your exterior door's security protect you from theft if it's your roommate who's stealing from you?

> No one is locking people up for political photos.

Authoritarian dictatorships like China are, and they'd love to use this tool to help them with that.

You have repeatedly misrepresented other people's viewpoints in order to win this argument. Maybe you have completely misunderstood what the discussion is about, but since you're doing it repeatedly and have been called on it multiple times, it is fair for others to assume bad faith.

Breaking encryption is not a "requirement" of a backdoor.

> Accessing unencrypted files on a device is not a backdoor

In what weird universe is this even remotely true?

The spirit of EFF comments is 100% correct. It got them a $100 donation from me today as well on top of my annual $500 donation at Christmas time.