The logic works both ways. Given a recognition system you should be able to build/train another system to find and amplify errors in the former. The more robust the former is the more likely the latter adversarial system is to be of lower complexity (cheaper) giving it some advantage.

The problem is that when you put a file out into the world, you can't continuously update it with the latest advances in adversarial machine learning. It'll have a reprieve ranging from months to years, then be vulnerable forever more.

But there's an upper limit since the photos still need to be recognizable to people.

I think the better word is “trained”, not “robust”.

By making almost invisible (to humans) distortions to a picture, a previously well-trained algorithm is now facing a completely unknown problem.

That implies the amount of training is the only factor. It may be that new architectures or training methodologies could be developed to reject large swathes of these types of advesarial examples. Though of course, if adversarial training is used avdersarial examples will obviously exist at some level, and so the arms race continues.

But for every adversarial distortion algorithm, you’d in theory have to double your training set, wouldn’t you?

I'm not sure I follow the point about robust facial recognition being cheaper to break. Like others pointed out, I don't see how this could be a scalable tactic.

It's an arms race then, sure. But why not stay in the race?

I don't think it is an arms race; that implies if you keep plpaying you're really just buying time and raising the stakes until one side ultimately wins all in a zero-sum game. SPOILER: the image classifiers will not only win in the end, all that defense you played is now fuel to make them even better.

I think a better notion is that of sunk cost. We have lost any approach that fights AI head-on. We should stop chasing this path and switch tactics, like changing online behaviours, education about privacy or, since developers love technical solutions, pursue technology solutions that leverage AI against this sort of cause. Example: what if we all switched to a morphing, deep-fake enabled visual representation? The better image identification got the more intense the msitakes would be.

The term we’re looking for might be “cat and mouse game.”

The last I looked, Jerry is still in the running.

I'm reminded of the suits they wore in A Scanner Darkly.

With the Purdue trial. I keep coming back to how this weird story was actually describing our reality.

The difference is that if you 'lose', all your previous victories change to losses as well, as all the images you've released that were previously impervious are now fair game.

That still doesn't mean it's worse than not doing anything at all.

It may, depending on the resources required. The thing about arms races is that they can easily impoverish everyone involved for no ultimate gain relative to one another.

Fair enough. I'd argue that in this case the winning move is not to make the potential liability of the data available.

As an 'arms race', the outcomes are very unbalanced.

it does mean doing the wrong this is worse than doing nothing. See: opportunity cost.

Legitimate point.

In the long run, you are dead, at which point you probably don't care as much.

Not necessarily. The phrase "buy some time" springs to mind. By the time your previous "wins" become "losses" it may not matter anymore.

The end of the arms race is seen in captchas — where to defeat computer recognition they are largely outside the capabilities of human recognition.

That is the nature of captchas by design. 2000s captchas can probably be solved by off the shelves OCR.

Because arms races are pointless.

See War Games (1983)

But WarGames is a fictional movie about a mutually assured destruction scenario? And even though it was famously watched by President Reagan [0], his response was decidedly not to abide by the philosophy of "the only winning move is not to play". Reagan presided over massive budget increases to keep up and attempt to win the arms race, including the extremely expensive but impotent "Star Wars" program [1].

Unlike "Star Wars", increasing the computational complexity and costs of mass face surveillance is not a decades-to-soon monolith project.

[0] https://en.wikipedia.org/wiki/WarGames#Influence

[1] https://www.armscontrol.org/act/2004_07-08/Reagan

I didn't watch the entire movie, just that clip- but I think that if you consider it slightly differently, the only winning move is not to launch, because everyone will die. If you don't have enough stuff to make sure everyone dies, then there is a chance of a winning move that results in some percentage of one party's population surviving and as a result they might take that chance.

IMO, As a U.S. President, Reagan made the right decision to keep pushing, especially considering that he knew to some extent that the Soviets, their enemies, couldn't keep up forever financially. It's a really really cold calculation, but one that worked out.

As we stand now, I think that privacy is on the losing side here, unfortunately- at the end of the day, we're up against our own and other governments, some of which have quite authoritarian regimes and have absolutely zero problems with at the very least using force to acquire funding.

In WarGames, the only way to win is "not to play" when the playing field is level- this playing field is not level- to be able to beat this off, we need to be able to level the playing field somehow. Maybe I'm not smart enough or just can't see something, but I frankly don't know how that would be possible.

...holy cow, that's dark.

Regarding the Star Wars program, perhaps the high cost was part of the motivation? (To saddle the opponent with similar costs)

I hear this quite often, but I just wonder if that is hindsight rewriting/spinning the situation? Did the people in charge of Star Wars program know it was a sham and knew from the beginning that's what it was, or were they really trying to build a space based defense program that turned out to be too complicated and too expensive to do?

In reality "star wars" was a moniker applied to several different programs which were part of the Strategic Defense Initiative geared towards missile defense. Part of the Strategic Defense Initiative was the ISTO, which funded basic research at national labs, which included some promising but ultimately impractical technologies - as is typical of basic research. Tech like isomer lasers and particle weapons made headlines, while missile guidance systems and radar systems and other such technologies which were much more seriously pursued didn't. Many of the technologies developed by the SDI remain at the core of modern missile defense systems. In fact, when one looks at the history in context, SDI only started getting wound down when the soviet union stopped being a threat and thus there was no longer a need to stop a large scale nuclear missile strike; while programs got renamed and were refocused on stopping different threats, the work never really stopped. So it would be inaccurate to describe the program as either a deliberate or accidental boondoggle.

That being said, ever since the very beginning of missile defense, it was always understood that the system would never be a permanently impenetrable shield. At best it might take a few years for adversaries to develop a new weapon to get through, but most likely a system would only be able to deal with a portion of targets. But the purpose of missile defense is not to raise the costs for the adversary - if now they have to launch 3 missiles to guarantee a hit where before they needed 1, you've essentially tripled their costs, not including the development of any new technology which could also be considerable. The fact is the Soviet Union did spend a substantial amount on developing technologies to defeat Star Wars, and even today China and Russia are spending significant resources developing technologies like hypersonic missiles to deal with the successor programs. So while the implication that the US purposely pursued technologies it knew would never work to trick the Soviet Union into following suit is not true, the US was trying to hurt its adversaries economically.

This is outright wrong. At some point one side outperforms, or "in universe(a la Cold War)" outspends/demoralizes the other and there is victory - aka, control of the global financial markets and geo-political dominance.

Is it a net win for humanity? Having stockpiles of nukes isn't. The space, science, athletic, and social welfare race is.

Competition drives progress. Hegemony leads to stagnation.

Ah, so mutually assured destruction has been impossible this whole time? What a relief!

I think you may be casually dismissing the entire field of game theory.

As far as I know, the elephant in the room problem [1] is still unsolved? Without support for double take, an adversary attack would still be fairly simple by simply including out of place information?

[1] https://www.quantamagazine.org/machine-learning-confronts-th...

> And in general, it's an unwinnable quest: by definition, you cannot have a system that makes photos "undetectable to facial recognition without ruining them", because if another human can recognize you on a photo, then a machine can be taught to do the same.

When I was about thirteen years old and in secondary school. For the information technology class there was a quæstion on the test which essentially amounted to answering whether machines would ever be able to perform any task a man can, and the “correct” answer was “no”.

I protested this and thankfully the teacher agreed that such speculation should not be part of the curriculum and that though unlikely, one can never truly know.

It's funny how a.i. has advanced so rapidly know that it's almost considered a certainty that for any given task, a machine can be made that does it as well as a man eventually,

The machines are just nibbling at the edges of what we can do. I think the jury is still out whether GAI will be a thing.

I am wondering if there are some powerful biases in the algorithms, for example "cats" so you add a few small shadows that a human will not notice and the ANN will trigger "Cat" all the time. Then if they fix this you need to find all the other triggers that are hypersensitive and focus on targeting those.

So you would add pixels in an image that trigger cat,car, bus, street etc . You will make the technology more expensive and harder to use by majority of bad actors.

> You will make the technology more expensive and harder to use by majority of bad actors.

I mean, we do have encryption, which is cheap for users and expensive for bad actors. The question is whether an analogue exists for AI.

In a sense, there's three types of security: exponential, proportional, and logarithmic. Encryption strength is exponential, number of pins in a lock is proportional, and bug fixing may be logarithmic.

What if we use DeepFakes to overwhelm the internet with pictures of people who don't exist? There by giving them LOTS of photos that have to be shifted through?

That would require that all social media is flooded with fake pictures (maybe more than it already is) making the Internet less useful for humans too.

> You will make the technology more expensive and harder to use by majority of bad actors.

It's software, mostly. People will share SOTA algorithms for free, like they do today. I don't think the cost for bad actors will rise much (and it's not like most of them have money problems).

On the contrary, I imagine it'll just accelerate development of computer vision that reproduces human vision in a bug-compatible way. The only way to confuse it will be to make the image confusing to humans too.

>It's software, mostly. People will share SOTA algorithms for free, like they do today. I don't think the cost for bad actors will rise much (and it's not like most of them have money problems).

Can I get access to high quality voice recognition or text to speech software? The only way I know is to use cloud APIs to get access to latest and greatest.

The algorithms, yes (see e.g. wav2letter and others), but not the training data. Mozilla's Common Voice (https://commonvoice.mozilla.org/en) is trying to make training data available, but proprietary systems can use that as well as the existing data advantage they have.

Most bad actors do have money problems, or at least are meaningfully resource-constrained.

The really scary ones, sure, they may as well have resources: Yes. But that's almost the biggest part of what makes them scary.

It could be very useful as a tool to avoid over-fitting though :)

Isn't this basically how GANs already work?

Not really. Well, sort of. If you have a small dataset, you can use data augmentation to "stretch" the dataset into a larger size. It's unknown whether data augmentation helps in general though (i.e. when you have lots of training data).

An augmentation is color dropout (photo becomes greyscale), cutmix (splicing two different training examples together), blur, noise, yada yada.

> if another human can recognize you on a photo, then a machine can be taught to do the same.

Sounds like:

If a human can do X then a machine can be taught to do the same.

Which is not necessarily true.

It's easy to say that a super generalised version of another person's claim isn't always true. But face recognition in static pictures works pretty well by now, and it would just take a little bit of digging to find examples where a trained model can do it better than a human.

What object in a picture can be detected just by a human but not a machine?

Trucks apparently: https://www.youtube.com/watch?v=X3hrKnv0dPQ

mirrors? not the reflection but the reflector.

> Which is not necessarily true.

Yet.

Humans are machines, so why wouldn't that be necessarily true?

Machines are designed, constructed, and built by humans.

If humans are "machines," then who built us?

I think it's debatable whether that's a necessary condition for something to be reasonably considered a machine.

I'm not saying this is an exact analogy to facial recognition, but there are all kinds of long-term unwinnable quests we participate in, because the alternative of giving up is worse: vaccines, antibacterial measures, crime prevention, fighting poverty, and so on.

There's clearly value in all of those things, even though we have no clear path to (or even defensible expectation of) ever completely solving any of them. In these cases we define progress as mitigation or delay of the consequences rather than elimination of them.

Some people can imagine a technological solution to them, but that's currently science fiction — a counterfactual — and is it more counterfactual to imagine a perfect adversarial image generator than it is to imagine a perfect antibacterial agent, or a post-scarcity economy? I don't think so, personally.

There's a difference between something we can't "win" but can suppress to low levels, and something that makes no difference.

This tool makes no difference. If it's deployed in any non-trivial capacity, anybody doing nefarious facial recognition will just train their recognizer against it.

These sorts of stories pop up every few months on HN about some tool that will defeat deep learning. But invariably they are also deep learning powered tools trained against a specific recognizer. The takeaway should be about how fragile current recognizers are, and not about any one tool that's going to "defeat" facial recognition.

It's like using a lock against thieves: you don't have do make an unbreakable lock, you just need to have a lock which is harder to crack than its neighbors.

Disagree. The reason that logic works with thieves is that the thieves have limited time and resources, and both high risk and high opportunity cost.

In this metaphor, if Thief #1 can't do it, he hands it over to a "bank" of 1,000,000,000 Thieves who will get to work opening it, while Thief #1 moves on to the next house.

Agree w gparent. The thieves largely use the same toolsets and aren't making bespoke tools for each job. If the obfuscation volume doesn't rise to a level where it is worth addressing by a standard toolset then the thieves won't have access.

After all the snakeoil that depends on facial recognition is bypassed, that will still before it's retrained.

Unless you can generate novel ways to do it continuously

That's a viable approach so long as three things are true. First, you can continue to do so in a way that your adversaries cannot follow your stream of novel changes. If they can follow they can adjust as you do and your novel ways fail upon arrival. Think Star Trek Borg.

Second, this approach works so long as novel ways can continuously be found and the systems you are attacking are continuously vulnerable to incremental attacks. In other words, this approach requires that the recognition you are attacking never generalize in any way beyond addressing your attacks. As others have noted, human brains can recognize past many kinds of distortion, including novels ones. This implies that computers can learn to do the same.

Third, and perhaps most concerning, this approach is only really useful if your adversaries never have, use, or find value in performing recognition on stored imagery. Stored imagery in the hands of your adversaries cannot be subject to ongoing novel distortions. It's fixed and can therefore be subject to recognition that has adapted at a later date. In short, for a novel approach to be useful it needs to stand up to attack for a sufficient amount of time as to not be useful for the attacker.

Using a continuous stream of novel distortions is an intriguing idea! It's a type of polymorphic attack. I think it's worth considering in the context of the traits that would make it most useful: an unpredictable and unfollowable stream of mutations, an un-generealizable approach, and resisting attack for long periods of time.

Also, Google isn't best in class at reverse image lookup so they chose to test themselves against the lower bar. If you try this exercise in yandex it finds results of Biden, you just have to crop out the surround text.

You need a machine to beat a machine.

Nah, I can always just pull the plug on the machine. Who wins now?