> I had my 70 year mother ask me last week if she should remove her voicemail message because can't people steal her voice with it? I was surprised but I guess she heard it on a Fox segment or something.

Out of curiosity, how much training data is needed currently to mimic a voice at various levels of convincingness?

Skype does it in real time (for live translation) with a few seconds of audio. For the reasons discussed in this thread, it continuously forgets its previous training from the call to not make the voice too similar, but just similar enough to distinguish the speakers.

Just a few seconds of speech gets you pretty far with a lot of models

Yes, this is generally true, in order to clone the timbre/pitch/tonality, but if you're going to run a voice cloning scam to fool friends and family members, you need much more to get the cadence, pauses, vocal tics, etc right.

Almost none, at most as little as a professional impersonator requires. GPT-4o's advanced voice mode would clone the user's voice by accident. A recording clip of one incident is available:

>During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice.

>Example of unintentional voice generation, model outbursts “No!” then begins continuing the sentence in a similar sounding voice to the red teamer’s voice

https://assets.ctfassets.net/kftzwdyauwt9/4CG0G7y9WOfEkzBpi7...

https://openai.com/index/gpt-4o-system-card/

This is idealistic. People still haven't fully learned that images can be photoshopped in its twenty years of its existence. (Deep)faked porn is still harmful which is why it's a crime.

Worse, there isn't an attitude of default skepticism in many areas/cultures. If a person is suspected of violating the moral code the priority will be punishment and reinforcing that such behavior isn't acceptable. Whether or not the specific person actually did the specific act is a secondary concern.

It's just going to increase the number of people who will be harmed or killed.

Yep, had a lawyer telling me that image timestamp cannot be faked. While it's literally a right-click away.

When people in positions of authority, like legal experts, don’t fully grasp how easily digital content can be manipulated

Well, thing is, most people can't.

In a lawyers view, and a judge's view, some skilled expert "hackers" can, and its called hacking. (so i guess we're all hackers)

I once discussed these things with a (knowledgeable) lawyer. He explained you can just present almost anhthing in a court case, and when it isn't refuted, well then it's valid.

In a case my lawyer (same one) presented a printed out email. Other party did not claim it was false, so it's suddenly just as valid as a registered letter. (it was a genuine email).

In another unrelated case, the other party suddenly introduced a forged picture. If I hadn't been there at that moment (I wasn't supposed to actually), then suddenly it would have been proof.

Court cases are not about truth, and not about justice. They are about convincing the judge.

And judge's job is not to find the truth. It's to convince the public that ruling was just.

Well, at least for email, it's theoretically possible to prove it's authenticity through third-parties. E.g. lawyer can ask GMail "did you receive this email with this DKIM?".

I'll note that both photoshop and changing the timestamp of images (mentioned below) are only easy for a very small percentage of the population. It'd likely be different if >30% of people could easily do these things.

Changing an image timestamp -- as in, exif metadata -- is trivially easy for anyone with a computer or phone, a quick search will tell you how to do it on any device with no skill involved. There's a difference between "easy for a small percentage of people" (modify the content of a photo in an undetectable way, find a software vulnerability, etc.) and "only a small percentage know how to do it" (modify exif values, do a simple magic trick, etc.). Just because someone doesn't know anyone can do it if they tried doesn't mean it's not trivial and pervasive.

The problem that to make a right query, one has to know at least part of the answer. To search for exif modification tool, it's essential to understand: - what is image file, and how information stored in it. - the concept of `metadata`, and exif in particular. - the concept that any file can be modified in place, or when you make a copy. - know what where's ecosystem of open/free/custom tools that can do an unusual tasks. The small percentage has all that knowledge at the hand to even get an idea of possibility to modify image timestamp

This.

I'm currently the originator and lead on a project that will have huge impact on our bottom line. The team is mostly junior people, so when I presented the plan the response I got wasn't "I didn't think we could do that", it was "I didn't even know that that is a thing".

There used to be a joke about Rumsfeld's "known unknowns" vs. "unknown unknowns", but it's so true. And to the vast majority of people in the world the question isn't "how" to change a timestamp, it's understanding that a timestamp can even be changed.

The first thing that should be done - which should be easy enough to do, is demonstrating blasphemy with the voices of all the most pious leaders that lead the lynch mobs. If those ones get believed, well, problem is solved in an alternate way.

I also however don’t believe that deepfaked porn is actually harmful. People are already imagining people naked, and it’s been easy to photoshop a nude for 15 years. A talented artist was needed 200 years ago. Allowing this (admittedly rude and disrespectful) act to be done with less skill changes very little.

Most people will believe a rumour if it is told to them in person by a friend. We've had our entire evolution worth of time to recognize that rumours can be manipulated yet rumours still spread and are still very dangerous.