IMO the creators should feel good about their actions, even if they feel bad or apprehensive about the direction of the world because this technology exists at all.
That's an argument I have a lot of time for, but it needs to be made, rather than - as at present - having the whole issue sidestepped. With any tool like this, I think there's a need for ethical due diligence as much as there is for engineering, and there's no evidence that this aspect has been considered at all.
Who you release things to, in what form, with what framing - all of these impact how something will be used. You can never eliminate bad actors, but you can encourage responsible use.
Probably same on macOS, but it's (mostly) not open source, so you can't really argue about it being an open source thing that lets you do evil stuff. You could argue about its availability though.
And it boils down to the point you (and others) were making: these are tools, the ethics are in how you use them.
[0] I know there are some forensic distros that come with anti-forensic tools. Still, Linux in general is the environment that allows that kind of thing to work in the first place, going back to the original point of whether it should be open source or not.
“Tool is expensive” is a real, effective deterrent to frivolous use of a tool. There’s a reason we pool resources into governments, which is explicitly so we (via a government) have capabilities that we (as individuals) don’t have.
That may not be a fair comparison. Nuclear weapons are difficult to make, and using them will not go undetected - the damage would be enormous. Deep fakes are a real threat today, and anyone willing to pay for the expertise (which isn't that hard to find) can have them made. By putting this out there, deepfakes are now something almost anyone can make which means the threat is imminent and needs to be addressed. I understand the argument, but still not sure about it. Just wanted to say it's different than nuclear weapons.
Now that the public knows broadly that digital manipulation is easy and they have seen it with their own eyes a ton things like that are believed less. Notice I do say less though, we still definitely have problems with misinformation being propagated with shitty-looking video/audio/photos that people should not believe in because they look/sound so inauthentic.
All that to say, these tools clearly exist. I think the public is better served by them being pervasive and well-known rather than having them be locked up in Hollywood studios, research labs, and government agencies.
It may be worthwhile to have someone create a bunch of deepfakes of politicians saying "This video has been faked" to drive awareness of this as a threat model.
E: Forgot the key bit, it would need to be spammed all over Facebook/Instagram/Mainstream social media.
In a perfect world, what should have happened was a separate team inventing a reasonably accurate deepfake detector to the world before the deepfake producer gained prominence (differential technological development), but interest, productivity and excitement on the researchers' part to release their work first seems to have foiled that.
It's honestly an interesting moral question.
We as a species have only survived ~75 years since we became aware of nuclear weapons, which is only a small portion of our overall history.
This has taught me to be mindful of the state of being unaware, because once you are aware of something it is impossible to reverse unless you're an amnesiac. Collectively forgetting something as a society then becomes infeasible without the technology and the willingness to revert our knowledge to a safer state, and a single outlier can ruin the entire system. When Jobs held up the first smartphone and we all started cheering and fantasizing, did any of us think it would lead to this?
(I also think that a pill to forget the experience of [addictive thing] would be revolutionary, but sadly our ingrained curiosity might undo the effect shortly afterward. A lot of tech takes advantage of our curiosity.)
This makes me wonder: what compels us to keep researching AI despite us also being able to hypothesize all these game-over scenarios? Why does it feel like we have no choice but to be crushed by unbounded progress?
Perhaps 50 years from we may find that nothing short of halting all research in the name of curiosity was the only option that could have preserved our culture/existence/values for another few decades.
Jobs didn’t hold up the first smartphone and smartphone proliferation has little to do with the AI research that led to deepfakes.
Do we aim for security through obscurity or work out how to deal with that increasingly terrifying reality? I think at some point we will be forced to do latter.
Maybe an interesting norm to try to generate in this community would be, “if you plan on publishing attack X, you should publish immediately alongside it your best guess as to the effective counter to X.”
In the case of nuclear weapons, one of those consequences is that poor countries are denied the ability to defend themselves.
This led to the bullying and invasion of countries without nuclear weapons by countries with nuclear weapons (most recently Russia invading Ukraine, but China and the US are equally guilty).
I'm reminded of a line from the TV show The West Wing, something like: "India must and will be a nuclear power. That way, we'll never be dictated to again."
I'm sure there are some very worthy technical efforts to improve mankind that are being hampered by the lack of easily accessible U-235 so why are we restricting this material from Chemists that want to Chem and Physicists that want to Fizz?
The OP specifically said tools "like this"
While it’s convenient to dismiss any discussion about A and something slightly similar to A as “slippery slope,” an inconvenient fact is that the slope often is slippery. It’s increasingly evident to anyone paying attention that information warfare can yield disastrous effects in a variety of ways: disrupting peaceful transition of power; disrupting critical infrastructure; disrupting command and control of ”traditional” WMD systems, even.
So that’s sort of the spirit of my question: how far does this argument go? What is the definition of “like this?”
An authorised firearms officer (AFO) is a British police officer who is authorised, and has been trained, to carry and use firearms. The designation is significant because in the United Kingdom most police officers do not routinely carry firearms, although they can be equipped with tasers. In 2019/20 fiscal year, there were 19,372 police operations throughout England and Wales in which the deployment of firearms was authorised and 6,518 firearms officers, 4.9% of the 132,467 active FTE officers. Following the November 2015 Paris attacks it was decided to significantly increase the numbers of armed officers, particularly in London.
We produced the software to run the drones. We didn't personally deploy them to kill those children.
We built software to guide the rockets. We didn't personally fire them at those hospitals.
We chose to startup our company providing profiles of suspects using scraped, publically available data to any government agency. What they do with the information isn't our problem.
We wrote the software, but it's not our fault or problem what happens because of who uses and because of how they use it. Our intent was good. The market and demand was there. What's wrong with providing a supply? How many software creators feel culpable? Vanishingly few. Who cares when comp is high?
It’s surprising how often this argument is made compared to, say, equal-opportunity access to fancy cars or good housing.
Seems like a good thing.
That isn't a defensive use!?
The cat is already out of the bag. Going "nooo, unpublish this" doesn't protect people from scammers, who will continue to use it.
Society must unlearn that seeing is believing. That is what is morally irresponsible, not this software.
Knee-jerk refusal to even consider thinking/talking about the use of technology is far more irresponsible than any other stance.
Even beyond that, if you're an authoritarian nation with a rather good lockdown on your news and internet, what's the downside of throwing tools like this at democracies?
In my view the age of tools like this has already been here for some time and they are going to their next step of evolution, and there is very little we can do about preventing the distribution of them without affecting other pieces of software and their distribution.
> Society must unlearn that seeing is believing.
A total lack of trust of our shared perception? Let me know how that works out. Might as well have everyone on acid 24/7.
It's no different from religion, which was the backbone of the western civilisation for quite some time.
For example, religion claims you have free well. For a materialist, any interesting definition of free will is impossible, because we are made of quantum particles with defined statistical behavior.
Of course, we cannot prove or disprove the existence of free will in other people either…
Videos aren't the only way to understand the world, and there was never any such thing as "direct observation" in the first place.
Now consider a few months after this hypothetical fake-and-refutation, what if Biden actually does announce an emergency threat to America, and people end up waiting a few days (or weeks, or months) for the refutation to come.
The “no such thing as trust” is not coming from this innovation in particular but rather the confluence of two general trends: 1) heavy reliance on digital media and 2) ever-increasing ability to fake digital media.
Note that you don’t even need large, successful falsified media campaigns to create distrust. All you need is for people to know that the capability exists, and trust is immediately damaged. Even trivially-detectable falsifications can end up being difficult to sort through if there are a billion fake pieces of information crowding out a hundred real pieces of information. This is true even if all billion fakes are trivially detectable by a human (which they won’t be).
I too dread a little all the mischief that's going to happen while the world learns about this. But people don't have to uncritically believe "the evidence of their own eyes", even though that has been very useful corrective to many mistakes, and will I guess be less so now. Evidence doesn't work that way: people always have to interpret everything in terms of their understanding of the world, and they can get better at that in response to changes like this.
The advent of widespread photo/video evidence was a big help for people trying to interpret was real and what wasn't. Then photography was knocked out of service by e.g. Photoshop (generally "easy photo manipulation"). Now video is getting knocked out of service as well. So, yes, it removes a useful tool in interpreting the world's evidence, and it's not clear what will replace that tool (if anything).
To be clear, the "no such thing as trust" was specifically a response to GP's assertion that people will "have to unlearn that seeing is believing." Okay, fine, but then what can you believe if you not even what you see? Again this specific tech doesn't get us all the way there, but it gets us closer.
There are ways to prove authenticity, e.g. with cryptographic signing.
People have been aware that lies can be typed for as long as we've had typing, but it still works. The fact that it still works a lot of the time hasn't ended the world. People are more aware that they can't trust a photo, yet people still fall for edited photos, and the world still turns. I don't think deep fakes are any different. Videos were not very trust worthy before deep fakes. Special effects have been a thing for many many decades. The world is still learning exactly how untrustworthy video is and there will always be people who fall for tricks and manipulations, but eventually the majority will adjust and society won't crumble.
We'll put our trust in what we see with our own eyes and in people and organizations who have proven themselves to be trustworthy. We'll worry about how real something is in proportion to what it would mean to us if it were real. When it really matters we'll trust experts to authenticate a photo or video as legitimate just like we do currently. If the world becomes less dependent on information presented in formats that can we know can be convincingly manipulated we'll all be better off. History tells us technology like this in the hands of the public won't cause people to abandon faith in everything and anything they see, or destroy civilization. We all just get a little more savvy, more industries will take advantage of the cool tech, lots of people will have fun and games with it, not much really changes for most people.
No one said it'll destroy civilization, don't worry.
Should that argument be made to the original research? I don't think it applies to one but not the other.
I admit that this limit is hard to define. It’s also extremely hard to enforce for pure data, if that turns out to be what we want to do.
I do think disclaimers like this are a little juvenile (it reeks of a US-ian litigation mentality), but you can easily imagine why they put it there. Perhaps instead of the author being less naive, you need to be more empathetic.
Personally, I like when people release proof of concepts, and hate it when they release weaponized tools. Especially when I inevitably end up reading reports where APT groups are using those tools.
I wouldn't term it a failure of empathy to ask that people consider the impact of their actions on others. I totally get why they have the disclaimer, and I wouldn't ask them to remove it, but I don't think it's enough. Given the clear potential for harm here, the potential uses and misuses of this tool should have been addressed directly.
People should be happy, that there are still white-hats who're reporting exploits, even for no profit. I personally switched to gray-market, can't take the shit we get from companies, anymore.
This tool released publicly is only to bring an awareness to the topic. Everybody else who needed to exploit this, have these tools developed in private.
That's a dangerous precedent. Would you apply the same logic to a kitchen knife? Or if for some reason only freemium products count (not sure why), then a pentesting tool?
I understand the underlying point you are trying to make but what are you proposing as an alternative exactly? Who gets to decide which products fall into a gray zone, whilst others are only for bad use. We already see this kind of shoddy thinking leading to keeping DALE-2 out of the public's hands (or at least that is their claim).
Regardless, I'm not proposing to ban either knives or this tool - I've been very clear about that. I do think that - with anything that has potential for harm - it's important for people to consider the possible consequences and to actually engage in the discussion about usage, rather than either washing their hands of the issue or declaring that any consideration will soon lead to arresting everyone for everything.
This is a path we've trodden countless times. With some things, we've collectively decided that no controls are necessary. With other things - poisons, nuclear weapons, a host of things depending on location - society has enacted controls of various efficacy and validity.
The responsible choice, regardless of whether you end up being for or any any given restriction for [thing], is to spend time thinking about it, discussing it, and - particularly when you've chosen to release something - acknowledging the potential issues.
I think I agree with everything you wrote except what I'm unfairly reading into this phrase: we don't need to and should not irrevocably "decide" once and for all. We might have been quite wrong to ban some poisons even in the situations that applied at the time when we banned them. If we were right to do so, times may have changed and we don't any longer. Even nuclear weapons perhaps, in certain situations.
Similar to a previous comment of mine in this thread, of course I don't know from your passing use of that word what you think about this! But I do think that that worldview, that these things are or should be "decided" in a permanent sort of way, is commonplace.
It's wordier, but more accurate, to say that I think society is (and should be) in a constant conversation with itself, deciding what to prohibit and to what extent. We regularly get this wrong, and we should never assume that our current collective position is unassailable; there should always be room for doubt and further discussion.
For instance, I once cheated by using gcc/godbolt to generate assembly output for a class from C code. By this logic, Richard Stallman should be blamed for my misconduct.
There are any number of reasons for supplanting another face onto your own, many of which are simply good fun. If you choose to use this for scamming or perverted reasons, so be it.
Moral posturing aside, perhaps there could be an invisible watermark or something included by default to easily identify less technically inclined actors as users of this tool.
The inventor of knives? The knife‘s manufacturer? The store who sold me the knife? I would say the responsibility lays 100% with myself.
I do not think it makes sense to pursue inventors for what happens to their creations, unless they actively encourage misuse.
Do guns kill people or do people kill people? Could be a relevant analogy.
https://www.npr.org/2022/06/03/1102755195/uvalde-special-ed-...
Sorry can't resist
Should McDonald's be banned? I do not think so. Anything can be misused.
To answer your question about what "good" uses there are for deep fakes: Cheaper and better special effects, available to a MUCH broader audience than before. I cannot even imagine all of the great use cases creative people will find for this.
Yes. I believe it should.
Regarding the deep fakes being good, I meant societal good. Not just fun tacky things that some people have a good time with. What social value does it bring to the world? We can do everything we need to without deep fakes. It has no value to the world.
Unjustified empirical claim.
And even if it were true, it would still be insufficient for justifying public disclosure, classic is-ought.
Also, only nerds stand up for security nerds.
I don't agree, but people will never reach consensus on this moral topic. So please don't call the other side "at best naive".
Importantly though, that aim is served by proofs-of-concept and information sharing more than it is by releasing point-and-shoot attacks. This comment puts it rather well [1].
As above, I'm not saying this shouldn't have been created, or even that it shouldn't have been released. I am saying that there are important ethical considerations and questions raised that should be addressed directly by the creators, rather than being handwaved away as in the quoted disclaimer.
https://openai.com/blog/better-language-models/
So individuals working on a technology like this have no clear societal consensus to use as a guide when making the decision. Strong arguments can be made in both directions.
There will always be people who find the very idea of thinking about things from an ethical perspective offensive; that doesn't mean they're right. A lot of valuable discussion in this space is drowned out by people shouting down attempts to explore nuance and shine a light on the grey areas [1].
I'm not asking for censorship or a government crackdown; I do think that it's irresponsible to dodge the issues raised, act as though they are settled, or dismiss any concerns as anti-progress. As you say, there is no clear societal consensus, and there are strong arguments in both directions; what I want to see is those discussions taking place, in good faith, and those issues being acknowledged, rather than ignored.
[1] https://twitter.com/emilymbender/status/1532699418281009154
What we're talking about (I think) is releasing something with a net upside, but with potential for misuse and the resulting liability?
Am I misunderstanding your point?
We can do that. We do it all the time, assigning varying levels of blame to different parties for different things.
Determining between proximal and ultimate causes, or assigning more weight to one cause or another, is not some impossible burden.
We're software developers, not kindergarten teachers.
How do you feel about Sqlite's code of conduct? Because if you don't like that, then you shouldn't be making moral statements in your own repositories.
This is also true of almost every other area of human effort, whether implicit or explicit. As a catch-all baseline, we have the various legal systems.
The fact that a significant proportion of the tech industry refuses point-blank to even discuss the idea of ethics is not a sign of superiority but immaturity.
My only point was: you can disclaim anything you want, but that won't stop people from suing you if they feel like it (this can happen anyway, even if you disclaim any and all responsibility) and creating something that can very easily be abused and then thrown out into the public even though it is standard practice in the security community may well end up being a boomerang.
There is some precedent for the creators of such software being sued:
https://knowtechie.com/a-company-that-makes-iphone-hacking-s...
So I would not be too quick to dismiss that this could happen to smaller entities or even open source creators of such tools.
https://www.nytimes.com/2022/02/15/nyregion/sandy-hook-famil...
That’s not remotely the same thing as holding a creator liable for the misuse of his work by unrelated third-parties.
Frankly, the illiberal push towards “ban everything that could be misused” is disheartening, especially when we see it applied to a speech-based tool like this one.
Also: note that the degree to which everything is labeled 'speech' is fairly uniquely American.
(Rightfully) labeling code as speech is what allowed us to break down that barrier.
Would you also be questioning the ethics of djb in 1995? https://en.wikipedia.org/wiki/Daniel_J._Bernstein#Bernstein_...
Because once created something will have a life of its own. The question is only meaningful prior to creating it. This goes for the atomic bomb, the invention of gunpowder, strong cryptography and any other tool.
But if you don't care about ethics then it is obvious why you would not ask that question to begin with and I think that technology without ethics is no better than your average arms manufacturer. Only there at least we have - in some cases - some control over the uses, whereas software multiplies at a speed that politics can not catch up with. This may be a good thing, or it may be a terrible thing, and if it is the latter you will only find out when it is too late if you don't spend some cycles thinking about it beforehand.
I liked the internet a lot better before I had to think about the security implications of every move I made, and the difference between 'bad actors' that are incapable of wrecking everything and 'bad actors' that are empowered by the brain children of the good actors far more than the good actors are (due to the asymmetry between destruction and creation) may be more than we can deal with.
Individual empowerment enables incredible creativity, engenders individual and societal resilience, and leaves space for those who do not fit the norm to contribute their talents in ways we might not otherwise anticipate.
The second — prior restraint to enforce “trust and safety” - requires exerting authoritarian or paternalistic control over others to achieve its aims. In the process, it gives those who prioritize power over others the tools they need to undeservedly secure it, and invariably leads to individual, social, and institutional ossification.
Having things go wrong is a good and necessary thing; a perfect world is a fantastically dull and oppressive one.
These views did not arise from a lack of ethical consideration; they were a very considered ethical foundation on which the internet was originally built. In many respects, it was originally built to stand in opposition (or at least offer an alternative) to the staid and oppressive institutions that prioritize trust and safety.
The main purpose as far as I have it down was to create a method for (government) researchers to communicate and to have a reliable communications network in wartime.
https://www.usg.edu/galileo/skills/unit07/internet07_02.phtm...
Feel free to rewrite history any way you want it though. Perhaps you were confusing the world-wide-web and the internet?
Gopher, Usenet, and IRC. Eventually, the web.
Yes, the internet grew out of government projects, but it was rooted in the openness of academic networks.
The “trust and safety” ethos is merely the staid hall monitors of the status quo doing their best to co-opt something that was only possible to create outside their influence.
Its healthy for them to lose, sometimes. Maybe even most of the time.
Opposing the “trust and safety” hall monitors is an ethical position, just one that’s (seemingly) entirely contrary to your own.
This logically does not compute.
I guess you mean in situations where guns are currently used, other strategies could be employed?
Since the goal of a firearm is to hurl a chunk of metal through the air at high speed, I can't really think of alternatives. Besides air rifles, we just have traditional powder guns filling that niche.
The usefulness of firearms extends beyond killing people, and I don't think "guns are literally designed for killing people" is very accurate. I think the majority of guns are not designed for killing people. I think the Ruger 10/22 is the best-selling gun of all time, and it's .22 caliber, which is definitely not what you'd choose if the goal of your design was to kill people.
Guns are universally classed as 'weapons' not as 'metal object throwers'.
Not really, not for the vast majority of them. For hunting or sport shooting a bolt action makes sense, but a 9mm pistol is a tool specifically designed to kill people, just like an automatic rifle is.
The Apache Foundation - great example, thank you - despite not creating weapons, has clearly put thought into how their work interacts with wider society and how to ensure positive outcomes as far as possible [1]. People absolutely don't have to agree on moral issues, but it is irresponsible not to have considered them.
No, it is naive to pretend facial recognition is worth anything when creating tools to defeat it is a mere academic exercise in reading papers and implementing algorithms described in those.
Don't shoot the messenger.
No. You don't.
I am closer to your line of thinking than not, but, at the same time, from where I sit, it seems that data monitoring went too far already and regular user has to have a way to circumvent it.
For that reason alone, this tool, even if it will result in some abuse, it just evens out the playing field.
irrelevant
We should do the exact opposite. Science and reason would cease to exist otherwise. Individuals wouldn't need to be held accountable, innovators/engineers/scientists/entrepreneurs would be. End of a free society.
Have you thought of the chilling effect this might bring?
I've been extremely clear throughout this comment section - I don't want censorship, I don't think this should be banned, and nowhere have I called for legal consequences for releasing this. I want people to think about their actions, and to discuss the ethical issues arising from them.
And yet you've jumped to calling me morally corrupt because I want to murder craftsmen. That's not a reasonable reading of my comments, or in any way a proportionate response.
If you want to talk about chilling effects and the importance of science and reason, how would you describe your comments? You're shouting down discussion with wild accusations.
Absolutely zero to do with you personally. You're building a strawman.
To you personally: If you're going around touting "other side's morality", you should be open to seeing what happens when this kind of thinking is adopted by the society. Don't play the victim game.
This isn't so much a strawman as a slippery slope made from them. It is not reasonable to leap from 'people should acknowledge and discuss the ethics of their actions' to 'blood-drenched tyranny'.
I am arguing that your case is fundamentally flawed, irrational, corrupt, inconsistent and dangerous.
Edit: I can't respond to your slippery slope fallacy but you started with an extremely broad stance "Anything that can be created, will be created. However, that doesn't free you from all moral culpability." I am responding in kind with an extremely broad implications. I guess we're busy debating which fallacy falls in whose arm with not addressing what I countered your argument with. First came the victimization, then a strawman card, then a slippery slope fallacy card. But you never considered responding to what happens when we take your ideas and consistently apply them. I still don't understand why you feel like you're being accused. I am arguing about the points you raised. No one is accussing you of anything.
If you want to argue against my case, then please do that, starting by engaging with the position I'm actually advancing [2]. It's difficult to discuss morality at all when you're deliberately engaging in violent rhetoric rather than with me.
[1] https://fallacyinlogic.com/slippery-slope-fallacy-definition... [2] https://news.ycombinator.com/newsguidelines.html
That way, the software could be used for pen testing, but it could cause a silent trigger to go off.
That could even be a way to monetize the software....
Make the tools available to everyone. As Elon musk says, sun is the best disinfectant.
Power however, is amoral and neutral
Anything that can be created, will be created. However, that doesn't free you from all moral culpability. If you create something, make it freely accessible and easy to use, then I think you are partly responsible for its misuse.
I'm not saying that they shouldn't have created this, or that they don't have the right to release it. But to create it, release it, and then pretend that any misuse was entirely separate to you is at best naive.