News for hackers. Stuff that matters.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by sva_ on Tuesday June 07, 2022 @07:43AM (#31652718)
To those who ask about the ethics of releasing something like this, I'd say that this technology already exists, and bad actors probably already can get access if they really want to and are sophisticated enough. Making this available to the general public will spread awareness of the existence of such tools, and can then possibly have a preventive effect.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by blagie on Tuesday June 07, 2022 @08:17AM (#31653067)
As someone with a stalker, I can't emphasize this enough. A stalker will go to all sorts of lengths to do bizarre shit. People don't believe it. I would guess governments will do some equivalent there-of.

Democratizing access to things -- including bad things -- has a preventative effect:

1) I can guard against things I know about

2) People take me seriously if something has been democratized

The worst-case scenario is if my stalker got her hands on something like deep fake technology before the police / prosecutor / jury didn't knew it existed. I'd probably be in jail by now if something like that ever happened. She's tried to frame me twice before. Fortunately, they were transparent. She'll try again.

Best case scenario is that no one has access to this stuff.

Worst case scenario is only a select group have access, and most people don't know about it.

Universal access is somewhere in between.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by RobertRoberts on Tuesday June 07, 2022 @09:42AM (#31654178)
I want to second this, as there are so many people that just simply can't believe how much time and energy some people will put into destroying someone else's life.

And when you ask for help, people think you are the insane one because they simply can't believe your story about the insanity of someone else.

I hope you find relief sometime from your stalker. I found it (not a stalker exactly) from letting the person burn themselves with their behavior so many times without me doing or saying anything in return to them (my strategy of non-direct conflict, and it worked for me), that eventually they ran out of people to manipulate and fool.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by Moru on Tuesday June 07, 2022 @12:22PM (#31656631)
A worse situation: What if you ARE insane, it's just that this time it's not your imagination running wild, it's actually someone stalking you? People that have some mental problem, whatever it is, have a much harder time since no one will believe them. It's always just "I know you are feeling bad right now..." even though the diagnose is ADHD or something else irrelevant.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by HWR_14 on Tuesday June 07, 2022 @10:44AM (#31655092)
Your "worst case" depends greatly on who the select group is. Is it movie studios making 9 figure budgets, or is it any obsessed person who can figure out how to find and install software.

Obviously, it's hard to imagine many situations like that, but you can imagine a process that required a 8-figure quantum supercomputer.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by shostack on Tuesday June 07, 2022 @11:58AM (#31656217)
This is too myopic a view. Think about how far graphics have come on mobile devices let alone high end gaming machines in the past 5 years. This technology will eventually be accessible in the palm of your hand via a powerful enough device or the cloud. It's more a question of when than if.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by HWR_14 on Tuesday June 07, 2022 @12:11PM (#31656466)
At that point the select group will change. It eventually reaches a tipping point, but usually it starts with a small enough select group
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by avereveard on Tuesday June 07, 2022 @12:42PM (#31656957)
history say you cannot control how selected the selected group is, no matter the implications, i.e. nuclear bomb secrets from manatthan project finding their way into russian hands

https://www.smithsonianmag.com/history/spies-who-spilled-ato...

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by HWR_14 on Tuesday June 07, 2022 @08:09PM (#31661633)
That's just not true. If you couldn't control the spread at all, some terrorist group would have already detonated one.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by blagie on Wednesday June 08, 2022 @08:07AM (#31666069)
I thought I had the knowledge to build a nuclear bomb when I was in high school. In retrospect, I'm not sure I was right, but I'm pretty sure I was close to right. From where I was, I certainly could have picked up the knowledge as an undergrad; it's no longer a hard problem with publicly-available information.

For better or worse, I did not have the requisite $2B-or-so to do so at the time. My allowance was $2/week. I don't think most terrorist groups have $2B either. Even if I did have the requisite $2B, I don't think I could have done so discreetly enough to not get caught; someone would have noticed a high school student buying up things like uranium or building appropriate facilities.

I'm more concerned about biological warfare. While I don't have the knowledge to engineer a super-virus, we're at the level where a few years intense study is all it takes to have that knowledge, and a few high school students do. The cost structure there is increasingly moving into the budget of high school students too. Things like DNA sequencing and synthesis have a Moore's Law-style curve, with falling prices.

(If this sounds unrealistic, I was in high school back in the days when nerds intensely learned physics; physics is much less interesting to nerds in 2022. Hot topics for nerds change over time, and we're out of the cold war and the space age. Today's high school nerds have moved on to today's hot topics like machine learning and microbiology)

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by Vecr on Tuesday June 07, 2022 @11:21PM (#31662813)
They could probably make one (it would probably fizzle though, but that's still nasty, even if not full theoretical power), they just don't have the highly enriched uranium to do it.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by avereveard on Wednesday June 08, 2022 @01:10AM (#31663472)
Knowledge spread and stealing one/building an enrichment facility are quite far apart.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by kmlx on Tuesday June 07, 2022 @01:19PM (#31657428)
> Democratizing access to things

not to be too pedantic, but this is not "democratizing access", as that would involve print-outs/usb sticks/discs of the code distributed to people that can't access the internet, accessibility issues, bias considerations etc etc. as such, this is just "access".

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by blagie on Tuesday June 07, 2022 @05:31PM (#31660336)
Athens, late medieval/early modern Poland, and the early US all had democratic systems of sorts where about 10% of the population, give-or-take, were in the electorate.

Likewise, slavery < serfdom < caste / Jim Crow / etc. class system < equality under the law with discrimination < equality

Access, inclusion, and democracy aren't binary concepts, and progress is gradual.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by bko on Tuesday June 07, 2022 @08:12AM (#31653005)
I agree with everything you said, but I we shouldn't deny that opportunistic bad actors don't exist. Or it might get on their radar and be exploited. Open source tools also tend to be better maintained, documented and reliable, so the bad guys will have a better tool.

That being said, bringing it to light also has benefits like you said. If the tool is out in the open and state of the art techniques are used, technology to detect its use will also benefit.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by belter on Tuesday June 07, 2022 @07:52AM (#31652804)
You are right. I saw some guy that looked like Matt Damon trying to sell me some crypto coins...
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by karmakurtisaani on Tuesday June 07, 2022 @08:00AM (#31652877)
It would be so ridiculously crazy that an established A-list actor would willingly promote those crypto scams. Must have been a deep fake!
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by HWR_14 on Tuesday June 07, 2022 @10:45AM (#31655103)
For what it's worth, actors seem to be securing themselves against this using IP rights.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by roughly on Tuesday June 07, 2022 @01:13PM (#31657356)
I'm reminded of Firesheep - https://en.wikipedia.org/wiki/Firesheep - which came out in 2010. It wrapped session hijacking on WiFi in an easily usable interface. The technique and the vulnerability wasn't anything new, but the extension raised awareness in a big way and really sparked a big push for getting SSL deployed, enabled, and defaulted everywhere.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by avivo on Tuesday June 07, 2022 @12:45PM (#31656993)
Ease of access does matter.

It only buys time, but that can provide the time needed to create countermeasures and ideally make those very accessible—somewhat similar to responsible vulnerability disclosure.

This piece goes into more detail: https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541... (excerpt from a working paper, part of which was presented at NeurIPs).

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by gambler on Tuesday June 07, 2022 @01:45PM (#31657777)
The entirety of deep fake technology was developed mostly in mainstream academia using "raising awareness" as an excuse. Paper after paper, model after model, repository after repository. Every single time the excuse was "if we don't do it, someone else will". This was going on for years and the explanation is absolutely laughable. Without countless human-hours put into this by academia, it's pretty obvious that this technology would be nowhere near its current state. Maybe some select military research agencies could develop something analogous. Currently this is accessible to literally every crook and prankster with internet access.

Also, the notion that "raising awareness" is going to prevent deep fakes from being used in practice shows complete and utter disconnect from reality. Most people who are skeptical are already aware how imminently fakeable all the media really is. Most people who still are unaware will remain so, no matter how many GitHub repositories some dipshits will publish.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by oliver910 on Tuesday June 07, 2022 @09:17AM (#31653792)
I agree that raising awareness that tools like this are possible is important and that sufficiently advanced actors can do this anyway, however I don't think in this case releasing pre-trained weights to the general public is responsible. This could probably be used to help bypass crypto exchange KYC for moneylaundering purposes. I'm not sure what the best access model is - email us with a good reason to get access to the weights perhaps - but what alarms me is there seems to be no consideration for misuse or responsible release at all.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by Karliss on Tuesday June 07, 2022 @10:09AM (#31654523)
Even without deepfakes any kind of system relying on a person (or computer) not being tricked by webcam video seems quite questionable. People could still be tricked with a spliced video fragments of the real person or makeup especially if the set of face expressions used during "liveness check" is known ahead of time.
Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by wussboy on Tuesday June 07, 2022 @10:55AM (#31655274)
I try to imagine how society will deal with this. What if deep fakes are so perfect that anyone can generate real-time footage of anyone else doing anything? As a society we’d need to move out of the virtual and back into the physical. Would that be such a bad thing?

I suspect this perfect deep fake technology might be a real boon to society.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by haswell on Tuesday June 07, 2022 @01:23PM (#31657489)
I’ve often wondered if it’s possible to put modern tech “back in the box”. At first, this seems impossible. It permeates our daily lives in so many ways.

Continued development towards a tech dystopia might be the only way. But I’m worried it will require the dystopia part before people will wake up and accept that moving back to the physical world might be required.

The next few decades will certainly be interesting, at the very least.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by verisimi on Tuesday June 07, 2022 @11:56AM (#31656180)
This technology certainly already exists and has probably been around for a long time.

https://youtu.be/CpAdOi1Vo5s?t=3786

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by light_hue_1 on Tuesday June 07, 2022 @11:21AM (#31655685)
That's like saying that nuclear weapons exist, and bad actors can potentially get them, so let's lower the bar so that anyone can.

Making such tools accessible is reprehensible. It will lead to more bad actors, to less trust in media and in any objective reality, and more erosion of our institutions and society.

There is absolutely no reason whatsoever for this. It's unethical and frankly downright evil.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by teakettle42 on Tuesday June 07, 2022 @12:03PM (#31656307)
On par with nuclear weapons? Downright evil?

You’re being absurd.

The technology exists, and it’ll get better. Pretending that it doesn’t exist or banning it won’t make it go away — it’ll just be used by the least scrupulous and most powerful.

Disruptive efforts like this are most upsetting to anxiety-ridden people who think that if they could just control things firmly enough, everyone and everything will be safe.

That kind of thinking doesn’t actually work, though, and it produces a stiflingly rigid, oppressive society that deserves to be upset occasionally.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by thingification on Tuesday June 07, 2022 @12:58PM (#31657164)
I think you describe the conventional wisdom well here.

But: are those really the only options? Paranoid stop-the-world wishful thinking or all-new-trends-are-inevitable?

I realise I'm almost certainly wildly misrepresenting your point of view with the latter label! I don't really mean to ascribe it to you. But I think the conversation often ends up tacitly as a debate between those two positions, both of which are mistakes. But one other option -- as an example -- is to find more complicated ways to stop doing <bad use of tech>.

A striking historical example to me because it's so old it can hardly be said to have been an invention that needed to be responded to -- rather just an apparent invitable fact of life: For the longest time, it seems everyday brutal violence on a level we find hard to imagine was common. At the time, that would have been obviously inevitable. Nothing else had ever happened. Then it basically stopped -- turns out it never was inevitable. Why? We didn't for example ban knives as a technology, but we did a lot of things over the centuries that, to a good numerical approximation, stopped us using them for violence (banning them in certain situations is only one, I suspect not the most important and maybe not even necessary).

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by haswell on Tuesday June 07, 2022 @01:36PM (#31657658)
I think in many ways, the continued evolution of technology will turn back the clock on progress in other areas of our lives.

In ages past, impersonating another person was not nearly as difficult as it is today. Advances in identity verification and such have eliminated a certain class of problems. The emergence of deepfakes doesn’t necessarily introduce “new” problems, it just resets the progress on some very old ones.

Assuming this is solved not by banning the tech but by making progress elsewhere, some pretty worrisome implications come with that. One antidote to deepfakes would be even more progress on identity verification and tracking the source of digital media for purposes of authentication / validity.

It’s not hard to imagine the negatives in such improvements at a time where we’re already tracked far too much.

I think a more ideal development would be a return to pre-tech habits. Meeting people in person. Less reliance on virtual communication, etc. But one need only look at the backlash against returning to the office to see that such a move would require something truly existential.

I suspect you are right in the long run, but I’m curious what other factors or forces might nullify the threat introduced by this tech in the short term, or if things really have to run off the rails before people decide that fundamental changes in how we interface with tech are required.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by thingification on Tuesday June 07, 2022 @02:02PM (#31657991)
> I think in many ways, the continued evolution of technology will turn back the clock on progress in other areas of our lives.

I don't disagree that it can have that effect.

Just to note: this use of "turn back the clock" is quite distinct from "turn back the clock on technology".

> Assuming this is solved not by banning the tech but by making progress elsewhere, some pretty worrisome implications come with that.

My point was not that we can make progress elsewhere to solve resulting problems (we can) but that we can also not use the technology in the "inevitable" bad way. It happens!

> One antidote to deepfakes would be even more progress on identity verification and tracking the source of digital media for purposes of authentication / validity.

One other antidote (of many I'm sure) would be to do less identity verification in the first place.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by verisimi on Tuesday June 07, 2022 @12:14PM (#31656497)
But who are the actors worse that the mafia that is already in control?

Institutions that are corrupt and serve themselves (not the public, despite their lipservice) need to go.

Re:Deepfake Offensive Toolkit (real-time deepfakes for virtu... by toss1 on Tuesday June 07, 2022 @02:08PM (#31658067)
>>It will lead to . . . less trust in media

Trust in media is already very low, and in fact should go lower. Deepfake tech exists, and the fact that it does, and is broadly available to bad actors, should be widely known.

Obviously the best case by far is that such weapons (in this case disinformation weapons) do not exist, but the worst case is them existing but hidden — THAT is the recipe for fooling people in the greatest numbers.

These tools existing, with widespread knowledge of their existence is sort of the least-worst case for the real world in which we live.

Yes this does have the potential to kill pretty much anything related to video and photography (everything from art to news to documentation), but that was the same when spam was a literal threat to existence of email. Unless we manage it, video and photography will be trusted for nothing but boring amusement; but better than than mass deception.