News for hackers. Stuff that matters.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by bamboozled on Monday January 09, 2023 @06:59AM (#34309376)
Oh that's great news \s

Do other people read these types of headings and just think immediately about how freaking scary this stuff is going to get...if it's actually accurate ?

Re:Microsoft’s new text-to-speech model can duplicate anyone... by dsr_ on Monday January 09, 2023 @07:10AM (#34309445)
Yes. Then I listened to the samples, and was relieved that it's not good enough to handle a well-known voice impersonation... yet.

My primary worry is for criminal misuse. There's a class of scams where the scammers find elderly victims, call them, and pretend to be their children or grandchildren on vacation in a plausible city and suddenly in need of money to help get their passport back or a ticket home.

There are any number of cases where I have accepted an order worth large sums of money over the phone from someone I have previously dealt with and feel familiar with their voice.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by rwmj on Monday January 09, 2023 @07:32AM (#34309614)
Only a year ago, my bank replaced their (already not very secure) second factor authentication with voice recognition.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by toomuchtodo on Monday January 09, 2023 @08:02AM (#34309853)
Re:Microsoft’s new text-to-speech model can duplicate anyone... by chinathrow on Monday January 09, 2023 @08:18AM (#34309978)
They will recycle it for sure once fraud hits their internal thresholds. Until that: self defense.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by SketchySeaBeast on Monday January 09, 2023 @09:02AM (#34310389)
It's mind boggling that the banks, who you think would have the most to gain from staying ahead of the tech scam curve, can't be bothered and instead continue to play a slow defense. I suppose they aren't losing THEIR money.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by silon42 on Monday January 09, 2023 @11:55AM (#34312506)
Time to look into robot-voice plugins that activate when the caller is not in the contacts.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by jacquesm on Monday January 09, 2023 @07:24AM (#34309545)
"Hi mom, it's Chrissy, I am calling you from someone else's phone, my phone got stolen. Do you have a pen handy, I'll wait. Ok, here is my new number, take the old one off your contacts right away, surely some scammer has it, and can you forward me 1000 bucks or so for a new phone, I'll pay it back when I have the new phone set up, here is my account number...."

When it's a text message it's likely going to ring a bell somewhere (hopefully an alarm bell), if it is a voice message and it sounds like the original many more people are going to fall for it.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by sendfoods on Monday January 09, 2023 @07:33AM (#34309617)
My god...I haven't really thought about scamming at all, I was "just" worried about manipulation of media and politics. New nightmare unlocked.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by hackerman123469 on Monday January 09, 2023 @08:12AM (#34309933)
"Okay, let me just video call you because I also need to show you something" and then the scammer hung up and was to never be seen again.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by probably_wrong on Monday January 09, 2023 @08:26AM (#34310033)
FYI, there's the following trick I read here some time ago:

The scammer starts a video call with the person they want to impersonate and record it. When that person takes the call, the scammer doesn't say anything. This creates a 5-10 seconds video of the person looking at the camera waiting for the scammer to say something, until they get fed up and hung up.

The scammer then calls the victim. They offer a video call as verification straight away, and they play that short video.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by Loughla on Monday January 09, 2023 @09:13AM (#34310504)
How is a 5-10 second video of someone either not saying anything, or saying, "Hello, who is it?" over and over convincing at all?
Re:Microsoft’s new text-to-speech model can duplicate anyone... by wccrawford on Monday January 09, 2023 @09:29AM (#34310683)
I imagine they say something like, "I couldn't see or here you. Did you see me?" through the other channel, and because modern tech messes up enough, it'll be believable to a lot of people. Especially the most gullible.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by citizenpaul on Monday January 09, 2023 @10:50AM (#34311650)
HN gets so caught up in logic. Has no one watched interviews of scammed people? Huge numbers of them report that they were suspicious it was a scam but they were so worried for a loved one they sent the money anyway because it was worth the risk to them just in case. Anything that helps push scams in the even slightly plailusible direction will make scams more successful.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by pictureofabear on Monday January 09, 2023 @08:27AM (#34310042)
All of these advances in technology have made the internet much less trustworthy. I wonder if we'll eventually hit a threshold where nothing is done virtually anymore because you can't trust it.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by sofixa on Monday January 09, 2023 @09:03AM (#34310404)
I think the work being done in the opposite direction is going to make it so the Internet can be used for high trust actions too.

E.g. in the EU we have ID cards and passports with biometric information and NFC, and there was a beta of a phone app by the French government where it would read your photo from the ID card/passport, and compare with a video selfie. That way you get fully local and secure identification allowing you to do stuff online that would otherwise require you to show up in person to a government office.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by jacquesm on Monday January 09, 2023 @10:34AM (#34311442)
That 'video selfie' is untrustworthy, which was sort of the point.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by sofixa on Monday January 09, 2023 @10:40AM (#34311514)
It's taken by the app directly, so it's somewhat hard to pass fake 3D video input to it.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by int_19h on Monday January 09, 2023 @04:56PM (#34316759)
Only if hardware is completely locked down.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by jacquesm on Monday January 09, 2023 @05:01PM (#34316828)
Scammers are well incentivized to find bypasses and feeding an alternative video stream into an app on a device that you control physically isn't all that hard for a determined person. I can think of three ways to do this right off the bat and if I think about it a little longer I'll probably find some more. I sincerely hope that that is not the last line of defense.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by jvm___ on Thursday January 12, 2023 @09:57PM (#34363441)
The open internet will be a dark forest, beautiful, but populated by metal-brains. Meat-brains will retreat inside trusted spaces below or above the empty forest.

https://maggieappleton.com/ai-dark-forest

Re:Microsoft’s new text-to-speech model can duplicate anyone... by teekert on Monday January 09, 2023 @07:30AM (#34309589)
So, we need to work on this ringing bells. Sure some people will fall for it initially, but then we adapt.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by Gigachad on Monday January 09, 2023 @07:39AM (#34309664)
Scammers have been scamming since the beginning of society
Re:Microsoft’s new text-to-speech model can duplicate anyone... by hef19898 on Monday January 09, 2023 @07:58AM (#34309811)
Oh, great! Now we have to come up with proper code words and code books for everyday life! I', not worried so, some AI solution will help us with that for sure.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by bambax on Monday January 09, 2023 @08:01AM (#34309845)
You can still

1. Ignore voice messages

2. Ask pointed questions to any live caller, esp. one calling from a new number and asking for money...

Re:Microsoft’s new text-to-speech model can duplicate anyone... by counttheforks on Monday January 09, 2023 @08:39AM (#34310151)
So now scammers will have to first scrape your & your families social media, then run that through ChatGPT to answer any questions.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by nullsense on Monday January 09, 2023 @09:07AM (#34310458)
Let the games begin.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by DonHopkins on Monday January 09, 2023 @10:30AM (#34311388)
The one earnest question that will ALWAYS make scammers hang up immediately:

"In case we get cut off, what phone number can I call you back on?"

If they actually give you a number instead of hanging up (which has NEVER happened in the many times I've used this line), just hang up immediately and call them back, to see if they answer.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by 4ggr0 on Monday January 09, 2023 @08:18AM (#34309973)
"Hey mom, could you tell me the name of your first pet, your favorite song and your birthday? Also, please fill out the captcha from this URL. xoxo chrissie"

EDIT: Also looking forward to AIs scamming each other, fully-automated.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by chirau on Monday January 09, 2023 @09:13AM (#34310509)
But AIs can't own crypto... or can they?
Re:Microsoft’s new text-to-speech model can duplicate anyone... by mikro2nd on Monday January 09, 2023 @09:55AM (#34310989)
What's to stop them? A wallet is just a keypair.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by 4ggr0 on Monday January 09, 2023 @10:05AM (#34311100)
Maybe they can bootstrap their own crypto? Or just open Monero wallets. If we're at the point where they can imitate voices and human behaviour, they might even pass KYC checks after stealing someones information.

As soon as AI gets out of hand and starts randomly hacking things we're probably out of luck anyways.

“If the rise of an all-powerful artificial intelligence is inevitable, well it stands to reason that when they take power, our digital overlords will punish those of us who did not help them get there. [...]” - Bertram Gilfoyle

Re:Microsoft’s new text-to-speech model can duplicate anyone... by pjc50 on Monday January 09, 2023 @12:02PM (#34312603)
I don't see what relevance that has other than making it easier to scam people irreversably?
Re:Microsoft’s new text-to-speech model can duplicate anyone... by wolpoli on Monday January 09, 2023 @05:48PM (#34317567)
They just need to earn their own crypto somehow. The AI could do then quite a bit even though they do not have Personhood, which could include lobbying politicians for Personhood.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by aceazzameen on Monday January 09, 2023 @08:47AM (#34310232)
This and who knows what else as time goes on. It might be a good idea to not talk on phones anymore unless you already know the contact.

In this bleak scrape the bottom of the barrel world, I can't help but think it's only a matter of time before customer service departments start selling voice data to the highest bidder. And to counter that there will be services where everyone ends up using voice changers, except close personal contacts. I want this fiction to stay fiction.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by MonkeyMalarky on Monday January 09, 2023 @09:26AM (#34310652)
>It might be a good idea to not talk on phones anymore unless you already know the contact.

As a millennial, I'm way ahead of you on that one. The government and telecom companies just need to crack down on number spoofing. Any day now..

Re:Microsoft’s new text-to-speech model can duplicate anyone... by 1over137 on Monday January 09, 2023 @09:08AM (#34310459)
>It might be a good idea to not talk on phones anymore unless you already know the contact.

As opposed to what? Writing is even easier to forge, no?

Re:Microsoft’s new text-to-speech model can duplicate anyone... by MisterTea on Monday January 09, 2023 @09:02AM (#34310397)
My grandmother was called by a scammer claiming they were me, said they were arrested and needed bail money or some nonsense. She knew it wasn't my voice, asked two questions which were dodged then told them where to go and how to get there. It left her very shaken and my mother had me call her to assure her I was fine and as soon as she heard my voice she bellowed "Now that's my grandsons voice!" With this tech all they need is a few seconds of my voice.

With AI we have lost all privacy and identity. This makes nonsense like novelty theory and timewave zero seem less nonsensical. Lets hope we make it through the great filter.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by nullsense on Monday January 09, 2023 @09:12AM (#34310501)
>Lets hope we make it through the great filter.

My thoughts on this are that oftentimes things at the extreme ends of the distribution are usually in some way, shape or form problematic. Compared to all other living creatures we know of, humans are right at the extreme tail end of the intelligence distribution, and the result is somewhat pathological when compared to other creatures. I think nothing that's north of a certain threshold required to make advanced technology can make it through the great filter without destroying itself in the process.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by galangalalgol on Monday January 09, 2023 @10:34AM (#34311439)
My thoughts are that getting noticed by another culture is the great filter, so cultures make it through occasionally, but we will not notice them. "The benefits of not being seen"

Of all my concerns for humanity, excessive intelligence is not one of them...

Our earliest (and easiest to detect) transmissions should already be far enough away to be far below ambient noise I think/hope. So maybe we made it through unless someone starts shouting. Own-goals are also certainly a thing, but it would take a serious mistake to end all of us, or even set us back that far technologically on a geologic time scale.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by nullsense on Monday January 09, 2023 @10:45AM (#34311580)
>Own-goals are also certainly a thing, but it would take a serious mistake to end all of us, or even set us back that far technologically on a geologic time scale.

I must be a pessimist in this regard. I don't necessarily think it would take a serious mistake, rather just the nature of complexity itself will likely become a factor at some point and I can imagine certain scenarios playing out based off that.

When you compare the length of time of humanity spent pre-neolithic-revolution with the short ~14,000 odd years since, I can't help but feel our current trajectory won't see us reach hundreds of thousands more years into the future without a serious backslide in that time.

Humanity is pretty darn resilient, but the last few years have highlighted to me that modern day life is some weird fantasy land we've created for ourselves that has an expiry date at some point in the future.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by bmacho on Monday January 09, 2023 @09:10AM (#34310482)
"Hi Chrissy, can you touch your ID to the phone so I can know that it is you" call ended

The government just should give you a card sized mini computer with 2 large primes stored in it, and everyone is perfectly fine. In fact, more secure, than the current situation?

Re:Microsoft’s new text-to-speech model can duplicate anyone... by addandsubtract on Monday January 09, 2023 @09:31AM (#34310705)
"My ID got stolen with my phone..."

It'd be better to ask something personal. "What did we do to Christmas/Birthday/[other event] last year? Who was with us?"

Re:Microsoft’s new text-to-speech model can duplicate anyone... by lm28469 on Monday January 09, 2023 @09:30AM (#34310686)
In the US where stealing a SSN is enough to impersonate someone I doubt the government would provide anything to mitigate these
Re:Microsoft’s new text-to-speech model can duplicate anyone... by galangalalgol on Monday January 09, 2023 @10:21AM (#34311268)
And telephone numbers can be spoofed for voice or text in the US. Not sure when the problem will get big enough that signed comms become normal. I guess imessage probably is? But only between users of apple devices.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by DonHopkins on Monday January 09, 2023 @10:28AM (#34311364)
Two large Prime minicomputers sounds pretty heavy for people to carry around with them all day!

https://en.wikipedia.org/wiki/Prime_Computer

https://en.wikipedia.org/wiki/Prime_Computer#/media/File:Pri...

Re:Microsoft’s new text-to-speech model can duplicate anyone... by mensetmanusman on Monday January 09, 2023 @09:15AM (#34310528)
Society will respond by having secret keywords that you know from a young age in your family to verify or to point to an emergency that is ongoing.

“ Here’s my new account number. It’s really me. Slippers in the toaster.”

Re:Microsoft’s new text-to-speech model can duplicate anyone... by dilawar on Monday January 09, 2023 @07:27AM (#34309570)
Recently in India, a politician deepfaked the voice to deliver his speech in multiple languages [1]. Can't comment on the quality of tech since i dont speak this language but it is pretty alarming to me. 2% vote swings make a huge difference in Indian politics since there are many regional parties. Almost more than 2/3rd of Indian voters didn't vote for modi yet his party is doing very well.

[1] https://www.theverge.com/2020/2/18/21142782/india-politician...

Re:Microsoft’s new text-to-speech model can duplicate anyone... by pjc50 on Monday January 09, 2023 @07:44AM (#34309702)
Interesting. Providing a translation is an accessibility measure, but it seems that the "fake" here is matching his lip movements to the translation. Almost not worthy of calling it a fake since he (not somebody else) did say those things (in a different language), and it's endorsed by him.

The unendorsed version is far more dangerous.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by thalassophobia on Monday January 09, 2023 @07:53AM (#34309769)
I think this is one of the cases where there is nothing wrong with using deepfakes. The person mimicked gave explicit consent and did not do any harm by that. If technology provides people increases accessibility of true political choice then it's good technology.

That said, I don't understand why the parent and the article paint this as a bad thing.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by jwestbury on Monday January 09, 2023 @08:31AM (#34310076)
When language is tightly coupled to cultural identity, using this kind of fake is lying to voters, saying, "I'm one of you."

It's a new use of technology to repeat an old lie. I think it's a bad thing, but it's hardly the most dangerous and novel application of this technology for evil.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by IgorPartola on Monday January 09, 2023 @08:43AM (#34310196)
Had he used a translator, it would have been more obvious but possibly had the same effect (if what he had to say resonated with the voters). So what you are against is the idea that it’s not labeled as “virtual translator was used to produce this speech”?
Re:Microsoft’s new text-to-speech model can duplicate anyone... by iso1631 on Monday January 09, 2023 @09:24AM (#34310620)
Certainly that's the impression I had from the comment - it's a politician being dishonest, and an undemocratic system which gives non-proportional power.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by piaste on Monday January 09, 2023 @08:50AM (#34310270)
Given that the PR firm who made their video apparently reached out to the press to tell the story, it seems that he wasn't actually claiming or pretending to be a native speaker (and anybody who did would be extremely easy to uncover). Just using a glorified overdub service and being quite upfront about it.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by mejutoco on Monday January 09, 2023 @08:22AM (#34309996)
The pope reads in phonetic alphabet to speak a big number of languages and most people do not consider it cheating. I do not se the problem.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by dharmach on Monday January 09, 2023 @08:45AM (#34310220)
>>Almost more than 2/3rd of Indian voters didn't vote for modi yet his party is doing very well.

Do you understand what is first-past-the-post voting system?

Re:Microsoft’s new text-to-speech model can duplicate anyone... by sendfoods on Monday January 09, 2023 @07:07AM (#34309422)
Yeah, I am increasingly nervous about some point in the future where you cannot trust any digital medium because of deep fakes, NERFs, voice cloning...

The only possible solution I have seen being talked about is digitally signing any and all "real" content, similar to GPG/SSH.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by ravivyas on Monday January 09, 2023 @07:16AM (#34309485)
Photoshop and minor video manipulation is enough to trick people into believing fake news, this will make it harder. I see political parties doing simple edits to make people believe anything, this will push it multiple notches higher.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by teekert on Monday January 09, 2023 @07:31AM (#34309597)
You really have low exceptions for humanity. Most people are not complete idiots you know. I think your opinion is pretty elitist. Or do you think you yourself will fall for this as well?
Re:Microsoft’s new text-to-speech model can duplicate anyone... by ravivyas on Monday January 09, 2023 @07:42AM (#34309686)
Less elitist(might really sound like one), more realist who knows that masses devolve to the lowest common denominator of sorts based on the information they are provided
Re:Microsoft’s new text-to-speech model can duplicate anyone... by citizenkeen on Monday January 09, 2023 @07:45AM (#34309706)
Not the person you’re responding to, but I don’t think it’s egotist at all. I’m sure I’ll fall for some news manipulation. I’m sure I already have.

I don’t think it’s elitist to think many people will be duped. I think it’s arrogant to think you won’t.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by pjc50 on Monday January 09, 2023 @07:45AM (#34309710)
Bunch of people just stormed the Brazilian government buildings yesterday over a fake "election theft" narrative. This stuff will be constantly tuned until enough people do fall for it, like advertising.

There's also the unravelling biography of https://en.wikipedia.org/wiki/George_Santos#Scandals (you know someone's doing well when their "scandals" section has ten subheadings! And that's all old-fashioned manual fraud)

(I'm now wondering how far you could get with an elected official who's entirely deepfaked and only appears in recordings. Probably all the way to being sworn in, at which point someone has to appear in person. People have occasionally been elected despite being dead, so it's not impossible)

Re:Microsoft’s new text-to-speech model can duplicate anyone... by swader999 on Monday January 09, 2023 @08:44AM (#34310207)
Or millions of people passively accepted the results of a stolen election. That would be an equally viable scenario with this technology. You just won't know the truth anymore.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by benlivengood on Monday January 09, 2023 @12:31PM (#34312995)
It's not equally viable; there are centuries worth of checks and balances designed to prevent voting fraud and errors vs. the ease of speaking a lie on video.

If electoral boards around the country or at the national level were all sounding the alarm for irregularity in voting returns or procedures then an election might have been stolen. As it stands, the election results match the earlier polls which is a strong indicator that, at best, the election became a tossup at the time it was held, and the military also didn't find any reason to suspect fraud or error.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by iso1631 on Monday January 09, 2023 @09:28AM (#34310665)
> Probably all the way to being sworn in, at which point someone has to appear in person. People have occasionally been elected despite being dead, so it's not impossible)

Unless the people doing the swearing in are also fake.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by willnonya on Monday January 09, 2023 @07:48AM (#34309738)
Perhaps most people aren't, evidence is pending, but enough of them have been proven to be that this is a real concern.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by nullsense on Monday January 09, 2023 @09:18AM (#34310563)
>enough of them have been proven to be that this is a real concern

This matters a loooot more than the smart enough people realize too, because it's so difficult to imagine what it's like being south of that threshold, the potential consequences are difficult to imagine.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by sethammons on Monday January 09, 2023 @08:00AM (#34309831)
Agent K: A person is smart. People are dumb, panicky dangerous animals and you know it. --Men in Black
Re:Microsoft’s new text-to-speech model can duplicate anyone... by timeon on Monday January 09, 2023 @08:27AM (#34310037)
It is easy to see through bullshit... ex-post. But not so easy in real-time.

> Or do you think you yourself will fall for this as well?

Yes.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by jacquesm on Monday January 09, 2023 @07:25AM (#34309554)
That they can do so without consequences is almost as scary as that they can do so in the first place.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by willnonya on Monday January 09, 2023 @07:50AM (#34309750)
Fraud, slander etc are already illegal. There are consequences for crossing these lines.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by jacquesm on Monday January 09, 2023 @07:57AM (#34309802)
AFAIK nothing ever happened to the political parties that use these tactics. But I'll be happy to be corrected.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by pjc50 on Monday January 09, 2023 @08:02AM (#34309849)
Not in politics they aren't. There was an incredibly high bar to clear for the Alex Jones libel lawsuit to succeed.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by iso1631 on Monday January 09, 2023 @09:27AM (#34310658)
> Photoshop and minor video manipulation is enough to trick people into believing fake news

If the news confirms what they think is the truth, it doesn't need manipulation, just a fake heading

https://www.snopes.com/fact-check/climate-protest-ukraine-bo...

for example (and in that case it even had the true caption on the video, just not in English)

Re:Microsoft’s new text-to-speech model can duplicate anyone... by hef19898 on Monday January 09, 2023 @08:00AM (#34309836)
I don't believe anything digital anymore for quite a while. Heck, pictures have been staged and faked since the beginning of photography. Modern tech just makes it easier.

Maybe, just maybe, human society will move back to personal interaction, since apparently you cannot trust call, voice and video, anymore in the future.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by pjc50 on Monday January 09, 2023 @08:13AM (#34309942)
> human society will move back to personal interaction

Which implies a much narrower, less diverse, more elitist system of both politics and economy, as it gets harder to trust people from outside your existing circle.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by hef19898 on Monday January 09, 2023 @08:18AM (#34309970)
True. Sometimes it seems some people read every CyberPunk story and considered it an instruction instead of something between a parody and a warning.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by brightstep on Monday January 09, 2023 @08:34AM (#34310111)
Hard to get more elitist than our politics and economy are now at the height of digital interconnection. The liberal democratic revolutions and popular democratic swell of the US labor movement under FDR all happened without the internet. A return of social connection to the community would be very much to our political benefit.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by teekert on Monday January 09, 2023 @07:30AM (#34309583)
When everyone know this is possible, we need better methods to verify what is real. We'll handle it, people said the same about photoshop.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by _visgean on Monday January 09, 2023 @07:36AM (#34309636)
Photoshop isnt realtime. This will be.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by swarnie on Monday January 09, 2023 @07:32AM (#34309607)
I suspect it is already widely known.

My employer is known for tracking a decade behind the curve and we already have financial processes in place to prevent this vector from being exploited. That leads me to believe the majority must already have as well.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by chmod775 on Monday January 09, 2023 @07:07AM (#34309423)
At some point natural people are going to use cryptographic signatures just to authenticate themselves in daily life. No video, audio, or text can be presumed real unless signed by the human subject.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by TeMPOraL on Monday January 09, 2023 @07:18AM (#34309505)
To do that, any meaningful communication will have to be recorded to be signed by the person, on a device provided to them by their Trust Authority company. Can't be just any device, because the chain of custody is as strong as its weakest link. Depending on the country, the Trust Authorities will be private companies with large regulatory exposure, like banks, or just governments themselves.

But hey, at least this will solve the problem of journalists quoting people out of context: they won't be able to claim you've said something without you personally signing off on it with your cryptographic key.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by acdha on Monday January 09, 2023 @07:30AM (#34309586)
There’d be a variable scale of trust: for many people, a signature from a generic phone would be fine since nobody is going to attack the secure element just to forge a valid signature. National level politics, or a billionaire’s divorce, could be a very different story, however.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by 323 on Monday January 09, 2023 @09:51AM (#34310929)
Or maybe it will be like today - if it's reported by Reuters/... it's trusted, if it's reported by DailyStar, maybe not.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by Asraelite on Monday January 09, 2023 @02:58PM (#34315163)
> Can't be just any device, because the chain of custody is as strong as its weakest link

I don't quite understand what you mean. We already trust phones and computers not to be compromised for things like online banking, so why not allow these devices to sign recordings using keys lent to them by the main trust device? You could allow revocation of these keys.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by TeMPOraL on Monday January 09, 2023 @04:50PM (#34316683)
> We already trust phones and computers not to be compromised for things like online banking

Sorta, kinda. "We already trust phones and computers" because we have no better alternatives yet.

There's a reason online banking is at the front of the push for device attestation, and thanks for Google happily obliging, rooting your Android phone is pretty much pointless now. This is coupled with a parallel push to do your banking through a mobile app, or at least to use it as a second authentication factor, which makes the app necessary just as much. This means we're already undergoing the transition I'm talking about: in the nearby future, you'll have to use a pristine, unrooted, unmodded phone from a blessed corporate vendor, to perform basic functions like paying for things and managing your account. And yes, such device will be appropriate for signing everything else too - and the vendor owning the key will be your Trust Authority.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by zigman1 on Monday January 09, 2023 @07:11AM (#34309449)
But... That's good no?

> No video, audio, or text can be presumed real unless signed by the human subject.

Where is the controversy in that?

Re:Microsoft’s new text-to-speech model can duplicate anyone... by jddj on Monday January 09, 2023 @07:17AM (#34309500)
Agreed. The time between now and then is much scarier.

Companies can't even wrap their heads around sim swapping scams and social engineering of secret questions.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by rippercushions on Monday January 09, 2023 @07:15AM (#34309477)
It would be nice if we were at 100% signing, but currently we're at ~0%, and the transition is going to be rough.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by willnonya on Monday January 09, 2023 @07:54AM (#34309781)
Why would that be nice?

Follow the thou g ht through to how it would work and what it would mean in terms of privacy for the average person.

Your niceties seem dystopian to me.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by Someone on Monday January 09, 2023 @07:47AM (#34309715)
Those subjects could lie.

Journalists would have a hard time showing that an interviewee actually said what claim they said.

Wire-tapping by the police also would become useless in court.

⇒ I would adjust it to “unless it has a reliably chain of trust”. That would mean that people could judge differently about whether to accept some things as true, but I think that’s unavoidable.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by lxgr on Monday January 09, 2023 @07:56AM (#34309800)
Given that digital signatures have been around for decades, yet banks, the government, and the legal system still treat signed (and potentially notarized) physical paper or scans of it as the gold standard, and SMS-OTP as a close second, I wouldn‘t hold my breath.

All digitally approved contracts I‘ve ever encountered were e-"signed" (i.e. me emulating a paper signature at a computer), not digitally signed, due to lack of a mutually trusted public key infrastructure.

In that sense, I would be sad but not surprised if we instead moved to a system of "trusted recording devices", where scanners, cameras and microphones by some vendors are defined to be trusted, signing their outputs and leaving the underlying workflows (signatures, recordings of verbal affirmations etc.) largely untouched.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by Semaphor on Monday January 09, 2023 @08:05AM (#34309870)
With all the (deserved) ridicule Germany gets for our failed digitalization efforts, we use public/private key pairs for our taxes, and IIRC even our national ID supports some PKI thing (not sure how that works exactly, never had any use for my ID’s online features, doesn’t seem widespread)
Re:Microsoft’s new text-to-speech model can duplicate anyone... by lxgr on Monday January 09, 2023 @10:02AM (#34311063)
> not sure how that works exactly, never had any use for my ID’s online features

I think that's a pretty apt summary of the state of digital ID in Germany: Commendable technical basis (with some privacy-preserving aspects too, including selective assertions, e.g. "older than 18" or "EU citizen").

When I called about actually getting such an ID card (several times) as a non-citizen, which is explicitly mentioned as a feature of the scheme, I only received the acoustic equivalent of blank stares.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by Semaphor on Monday January 09, 2023 @10:23AM (#34311284)
When my wife got her temporary residence permit in 2018, it came with exactly the same online features as our national ID card. Yours might have been too old.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by lxgr on Monday January 09, 2023 @01:24PM (#34313935)
EU citizens don't need (and in fact can't get) a permanent residency card.

For that reason, Germany introduced a "non-identity" (i.e. not doubling as a photo ID) e-signature card in 2021, but apparently almost nobody is requesting it, so most administrative offices don't know how to issue one...

Re:Microsoft’s new text-to-speech model can duplicate anyone... by Semaphor on Monday January 09, 2023 @01:44PM (#34314246)
Oh wow, that sounds like quite the oversight. I’m fully of the opinion that my fellow Europeans should be allowed to have a digital ID that they can’t use anywhere ;)
Re:Microsoft’s new text-to-speech model can duplicate anyone... by pjc50 on Monday January 09, 2023 @07:56AM (#34309797)
People talk about cryptography, which has fairly terrible adoption rates outside of invisible systems like TLS, but in practice the way humans authenticate sources is heuristic: do I trust the person telling me this? Does it agree with things I already know?

That is, you have a "filter bubble" which tells you that there isn't going to be a Nigerian prince with some money waiting for you (unknown sender, unlikely message). Reduced cost fakes force people to make their filter bubble smaller by raising the threshold for faking. More broadly, the question of "high trust vs. low trust" society.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by worldsavior on Monday January 09, 2023 @08:04AM (#34309861)
No because I know as technology advances, security advances.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by mrtksn on Monday January 09, 2023 @08:14AM (#34309946)
With all that ML stuff going on we can no longer trust our devices to guarantee authenticity of their behaviour. By that, I mean we can no longer be sure that the human voice produced by the speaker of our handheld computers is happening because a human somewhere else spoke to a microphone.

Unfortunately the move towards it is nothing new, we gradually lost text and images and now the remaining media is closing.

This will probably decimate online public forums completely and off-person communications will en up being P2P through trusted devices. Closed communities in WhatsApp, Telegram, Slack and Discord already replaced forums.

Maybe if infestation of closed communities with machines becomes a thing, maybe the real life gatherings will make a comeback?

I don't know but I used to mock Sci-Fi movies about portraying aliens as uncivilised that don't seem to have the means to have developed all that tech but I'm no longer sure. Maybe what will happen is, we will create a symbiotic life where machine will no longer be tools but partners and they will take care of us to reduce us to our basic instinct.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by tootie on Monday January 09, 2023 @09:05AM (#34310428)
A company called Lyrebird claimed they could replicate a voice in less time and had demos available a few years ago. The deepfake scare of 2019 also seems like a long time ago.
Re:Microsoft’s new text-to-speech model can duplicate anyone... by lannisterstark on Monday January 09, 2023 @10:44AM (#34311553)
Oh great, more fearmongering.

What cases do you think it will be used wrongly in? It can't do 1:1 realistic phone calls, the processing it'd need for that is beyond the reach of any average user.

I just would like this so I can copy-paste books/text and listen to it in the voices of my favorite narrators, but can't have that, can we? No, it'll be super dangerous, I'll listen myself to nuking a country or some such.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by r-zip on Monday January 09, 2023 @11:26AM (#34312137)
If you seriously don’t think this will be widely abused, then you’re extremely optimistic.

Spreading propaganda is the most obvious use case, assuming the resource requirement remains high. I don’t think it’s far-fetched either. If these models become less resource-hungry, then we’re in for much worse.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by Tao3300 on Monday January 09, 2023 @11:34AM (#34312250)
> the voices of my favorite narrators

If you really like them and their work, why would you want a forgery?

Re:Microsoft’s new text-to-speech model can duplicate anyone... by alfalfasprout on Monday January 09, 2023 @03:28PM (#34315536)
Compute keeps getting cheaper. This will absolutely be within the reach of a dedicated user in a very short span of time.

If you can't see the danger in this, perhaps it's more that you're ok with it.

Re:Microsoft’s new text-to-speech model can duplicate anyone... by XorNot on Monday January 09, 2023 @05:52PM (#34317647)
No, I read these types of headings and think how awesome open world gaming is going to be when any random developer can fully voice their game cheaply. The AAA benefits too: you get a lot of stilted dialogue when the script and gameplay diverge and you have to work with what you have - a world where the writers can just redo that on the fly pretty much up till release would mean we'd be looking at a new era of dynamic storytelling.

It'll be interesting when someone releases the first MMO which combines GPT models and VO generation to fill out the in-universe world with dynamic characters who can react to surrounding events.