That's an argument I have a lot of time for, but it needs to be made, rather than - as at present - having the whole issue sidestepped. With any tool like this, I think there's a need for ethical due diligence as much as there is for engineering, and there's no evidence that this aspect has been considered at all.
Who you release things to, in what form, with what framing - all of these impact how something will be used. You can never eliminate bad actors, but you can encourage responsible use.
Probably same on macOS, but it's (mostly) not open source, so you can't really argue about it being an open source thing that lets you do evil stuff. You could argue about its availability though.
And it boils down to the point you (and others) were making: these are tools, the ethics are in how you use them.
[0] I know there are some forensic distros that come with anti-forensic tools. Still, Linux in general is the environment that allows that kind of thing to work in the first place, going back to the original point of whether it should be open source or not.
“Tool is expensive” is a real, effective deterrent to frivolous use of a tool. There’s a reason we pool resources into governments, which is explicitly so we (via a government) have capabilities that we (as individuals) don’t have.
That may not be a fair comparison. Nuclear weapons are difficult to make, and using them will not go undetected - the damage would be enormous. Deep fakes are a real threat today, and anyone willing to pay for the expertise (which isn't that hard to find) can have them made. By putting this out there, deepfakes are now something almost anyone can make which means the threat is imminent and needs to be addressed. I understand the argument, but still not sure about it. Just wanted to say it's different than nuclear weapons.
Now that the public knows broadly that digital manipulation is easy and they have seen it with their own eyes a ton things like that are believed less. Notice I do say less though, we still definitely have problems with misinformation being propagated with shitty-looking video/audio/photos that people should not believe in because they look/sound so inauthentic.
All that to say, these tools clearly exist. I think the public is better served by them being pervasive and well-known rather than having them be locked up in Hollywood studios, research labs, and government agencies.
It may be worthwhile to have someone create a bunch of deepfakes of politicians saying "This video has been faked" to drive awareness of this as a threat model.
E: Forgot the key bit, it would need to be spammed all over Facebook/Instagram/Mainstream social media.
In a perfect world, what should have happened was a separate team inventing a reasonably accurate deepfake detector to the world before the deepfake producer gained prominence (differential technological development), but interest, productivity and excitement on the researchers' part to release their work first seems to have foiled that.
It's honestly an interesting moral question.
We as a species have only survived ~75 years since we became aware of nuclear weapons, which is only a small portion of our overall history.
This has taught me to be mindful of the state of being unaware, because once you are aware of something it is impossible to reverse unless you're an amnesiac. Collectively forgetting something as a society then becomes infeasible without the technology and the willingness to revert our knowledge to a safer state, and a single outlier can ruin the entire system. When Jobs held up the first smartphone and we all started cheering and fantasizing, did any of us think it would lead to this?
(I also think that a pill to forget the experience of [addictive thing] would be revolutionary, but sadly our ingrained curiosity might undo the effect shortly afterward. A lot of tech takes advantage of our curiosity.)
This makes me wonder: what compels us to keep researching AI despite us also being able to hypothesize all these game-over scenarios? Why does it feel like we have no choice but to be crushed by unbounded progress?
Perhaps 50 years from we may find that nothing short of halting all research in the name of curiosity was the only option that could have preserved our culture/existence/values for another few decades.
Jobs didn’t hold up the first smartphone and smartphone proliferation has little to do with the AI research that led to deepfakes.
Do we aim for security through obscurity or work out how to deal with that increasingly terrifying reality? I think at some point we will be forced to do latter.
Maybe an interesting norm to try to generate in this community would be, “if you plan on publishing attack X, you should publish immediately alongside it your best guess as to the effective counter to X.”
In the case of nuclear weapons, one of those consequences is that poor countries are denied the ability to defend themselves.
This led to the bullying and invasion of countries without nuclear weapons by countries with nuclear weapons (most recently Russia invading Ukraine, but China and the US are equally guilty).
I'm reminded of a line from the TV show The West Wing, something like: "India must and will be a nuclear power. That way, we'll never be dictated to again."
I'm sure there are some very worthy technical efforts to improve mankind that are being hampered by the lack of easily accessible U-235 so why are we restricting this material from Chemists that want to Chem and Physicists that want to Fizz?
The OP specifically said tools "like this"
While it’s convenient to dismiss any discussion about A and something slightly similar to A as “slippery slope,” an inconvenient fact is that the slope often is slippery. It’s increasingly evident to anyone paying attention that information warfare can yield disastrous effects in a variety of ways: disrupting peaceful transition of power; disrupting critical infrastructure; disrupting command and control of ”traditional” WMD systems, even.
So that’s sort of the spirit of my question: how far does this argument go? What is the definition of “like this?”
An authorised firearms officer (AFO) is a British police officer who is authorised, and has been trained, to carry and use firearms. The designation is significant because in the United Kingdom most police officers do not routinely carry firearms, although they can be equipped with tasers. In 2019/20 fiscal year, there were 19,372 police operations throughout England and Wales in which the deployment of firearms was authorised and 6,518 firearms officers, 4.9% of the 132,467 active FTE officers. Following the November 2015 Paris attacks it was decided to significantly increase the numbers of armed officers, particularly in London.
We produced the software to run the drones. We didn't personally deploy them to kill those children.
We built software to guide the rockets. We didn't personally fire them at those hospitals.
We chose to startup our company providing profiles of suspects using scraped, publically available data to any government agency. What they do with the information isn't our problem.
We wrote the software, but it's not our fault or problem what happens because of who uses and because of how they use it. Our intent was good. The market and demand was there. What's wrong with providing a supply? How many software creators feel culpable? Vanishingly few. Who cares when comp is high?
It’s surprising how often this argument is made compared to, say, equal-opportunity access to fancy cars or good housing.
Seems like a good thing.
That isn't a defensive use!?
The cat is already out of the bag. Going "nooo, unpublish this" doesn't protect people from scammers, who will continue to use it.
Society must unlearn that seeing is believing. That is what is morally irresponsible, not this software.
Knee-jerk refusal to even consider thinking/talking about the use of technology is far more irresponsible than any other stance.
Even beyond that, if you're an authoritarian nation with a rather good lockdown on your news and internet, what's the downside of throwing tools like this at democracies?
In my view the age of tools like this has already been here for some time and they are going to their next step of evolution, and there is very little we can do about preventing the distribution of them without affecting other pieces of software and their distribution.
> Society must unlearn that seeing is believing.
A total lack of trust of our shared perception? Let me know how that works out. Might as well have everyone on acid 24/7.
It's no different from religion, which was the backbone of the western civilisation for quite some time.
For example, religion claims you have free well. For a materialist, any interesting definition of free will is impossible, because we are made of quantum particles with defined statistical behavior.
Of course, we cannot prove or disprove the existence of free will in other people either…
Videos aren't the only way to understand the world, and there was never any such thing as "direct observation" in the first place.
Now consider a few months after this hypothetical fake-and-refutation, what if Biden actually does announce an emergency threat to America, and people end up waiting a few days (or weeks, or months) for the refutation to come.
The “no such thing as trust” is not coming from this innovation in particular but rather the confluence of two general trends: 1) heavy reliance on digital media and 2) ever-increasing ability to fake digital media.
Note that you don’t even need large, successful falsified media campaigns to create distrust. All you need is for people to know that the capability exists, and trust is immediately damaged. Even trivially-detectable falsifications can end up being difficult to sort through if there are a billion fake pieces of information crowding out a hundred real pieces of information. This is true even if all billion fakes are trivially detectable by a human (which they won’t be).
I too dread a little all the mischief that's going to happen while the world learns about this. But people don't have to uncritically believe "the evidence of their own eyes", even though that has been very useful corrective to many mistakes, and will I guess be less so now. Evidence doesn't work that way: people always have to interpret everything in terms of their understanding of the world, and they can get better at that in response to changes like this.
The advent of widespread photo/video evidence was a big help for people trying to interpret was real and what wasn't. Then photography was knocked out of service by e.g. Photoshop (generally "easy photo manipulation"). Now video is getting knocked out of service as well. So, yes, it removes a useful tool in interpreting the world's evidence, and it's not clear what will replace that tool (if anything).
To be clear, the "no such thing as trust" was specifically a response to GP's assertion that people will "have to unlearn that seeing is believing." Okay, fine, but then what can you believe if you not even what you see? Again this specific tech doesn't get us all the way there, but it gets us closer.
There are ways to prove authenticity, e.g. with cryptographic signing.
People have been aware that lies can be typed for as long as we've had typing, but it still works. The fact that it still works a lot of the time hasn't ended the world. People are more aware that they can't trust a photo, yet people still fall for edited photos, and the world still turns. I don't think deep fakes are any different. Videos were not very trust worthy before deep fakes. Special effects have been a thing for many many decades. The world is still learning exactly how untrustworthy video is and there will always be people who fall for tricks and manipulations, but eventually the majority will adjust and society won't crumble.
We'll put our trust in what we see with our own eyes and in people and organizations who have proven themselves to be trustworthy. We'll worry about how real something is in proportion to what it would mean to us if it were real. When it really matters we'll trust experts to authenticate a photo or video as legitimate just like we do currently. If the world becomes less dependent on information presented in formats that can we know can be convincingly manipulated we'll all be better off. History tells us technology like this in the hands of the public won't cause people to abandon faith in everything and anything they see, or destroy civilization. We all just get a little more savvy, more industries will take advantage of the cool tech, lots of people will have fun and games with it, not much really changes for most people.
No one said it'll destroy civilization, don't worry.
Should that argument be made to the original research? I don't think it applies to one but not the other.
I admit that this limit is hard to define. It’s also extremely hard to enforce for pure data, if that turns out to be what we want to do.
IMO the creators should feel good about their actions, even if they feel bad or apprehensive about the direction of the world because this technology exists at all.