What is in this demo is a very rate-limited, early version of our new model. We have many mitigations in place to increase the safety of our main product (Play.ht); I mentioned some of that here https://news.ycombinator.com/item?id=35331310

Then why release this outdated version?

The mitigations don't seem to address targeted attacks that I'd think you can assume will happen.

How do you address the civil and criminal liability of that?

Are there even any criminal laws against this? I mean, it strikes me that there should be, but my non lawyerly self has never heard of any.

Yes, absolutely. Using one‘s likeness without explicit consent first is illegal in (most of?) Europe and is a tricky subject in the US. For example look at Crispin Glover‘s lawsuit against Back to the Future II.

https://en.wikipedia.org/wiki/Personality_rights

To be clear, I'm looking for references to criminal law, not civil or case law. This all seems a combination of the latter, and even here it's not obvious what applies in cases where a third party produces the infringing content.

I am not a lawyer.

I had to briefly look up the difference between criminal law and case law to see what you mean. I have no idea if there are any criminal cases in the US about this.

In Civil Law countries as opposed to Common Law countries the plaintiff could very well be the state for this type of legislation. For example, look at tech companies being fined for GDPR violations, same basic idea.

Your mitigations are so bad, you must know that they are almost useless. No website has ever kept up with spam by individually checking every complaint, and you would only stop people after they've already recorded large amounts of output from a stolen voice.

On the other hand, as you also know, you could easily put a much stronger safeguard on by making people say a short prepared statement and only clone voices who have said that statement. You do not appear to have done that. Why not, except it would make it harder to voice steal?

Moral is the last thing YC cared about . In fact Paul wrote an essay about good founders being the kind of people who tend to break rules. Rules, laws , morality are for other people

I think it's impossible to write anything with nuance on the Internet, as it is always taken out of context and used as a gross caricature of the original point, as you have done.

I'm quite familiar with pg's essay, and the idea that he's arguing that "Rules, laws , morality are for other people" is laughable. Sure, it's fair to argue against his point, but one of the main thing he highlights in his essay is clearly knowing the difference between rules and laws that are morally important, versus ones that came about through regulatory capture or "tradition".

Most societies have at least some tradition of celebrating "good trouble".

I feel silly for saying this, but it's very obvious that rule breaking is not the same as acting immorally.

Machines should be able to speak in realistic voices. Every cool future I can imagine includes that. Why muzzle it just because people can be scammed with it?? People get scammed by jpgs, random people calling and saying they're from the bank or the IRS. I'm really not interested in limiting what humanity can accomplish to account for folks getting scammed. Many of them will get scammed anyway. What we should do is make it easier for these individuals to learn how to not get scammed or to come up with scam insurance for the ridiculously gullible. I'm just spitballing, but banning cool technology because people can be scammed is overkill.

My problem isn't computers speaking in realistic voices. The problem is speaking in your voice, or my voice.

Just because there's some bad stuff out there doesn't mean that opening the flood gates to higher amounts of more advanced bad stuff is acceptable. It's absolutely reasonable to look at a new technology and try to figure out if it is likely to bring about more harm than good and take action based on that assessment.

That's not fair at all. YC has declined to invest in many startups that they didn't think were doing good things. I don't know anything about the thought process in this particular case but you're miles off base with "Moral is the last thing YC cared about".

I don’t want to topple US democracy. I just want to make walk through videos for my apps without my terrible accent coming through :)

So thanks for funding this.

The constant worry about malicious deployment is so tired in my opinion. The technology to clone voices exists. Your trust is audio recording should already be shaken. Trying to hobble this product on the grounds of "it's dangerous" just serves to limit creativity.

I think this in an area where there are many more malicious use cases than legitimate.

It's like spyware developers that claim their software is for remote administration of computers you own.

Don’t take this personally, but I don’t think you’ve thought too hard about what you can do with this technology. For instance, you could create audiobooks for every book ever published. You can change scripts for movies after shooting has already happened. Indie game developers can now afford high quality voices in their games. Even AAA games like The Elder Scrolls can vastly expand their in-game voice variety. I think it’s amazing.

You're not wrong, all of that is great. But the capacity for even more spam calls and scammers generating fraudulent content with cloned voice samples will be an immensely annoying issue. Anyone who owns a phone in the modern age absolutely cannot be a Pollyanna when looking at this technology. There are real issues that must be addressed.

Is access to realistic voice really the limiting factor in spam calling?

Right now if your brother texts you and says they are stuck somewhere and need you to wire them $500, you call them and verify it is legitimate. That is the best answer we have for users.

Once your brothers voice has been cloned and generative AI can deep fake his face on a video call, we are pretty much screwed.

I usually just make sure the number is right.

How would you do that if they're claiming they lost their phone and are thus calling for a pay phone or the phone in a police station?

asking someone to take basic precautions like “have a fire extinguisher” or “hazmat labeling” just gets in the way of innovation!

well no, their argument is more like: "Everyone should keep advancing the art of cooking, but just make sure they have a fire extinguisher handy" and your response is "Don't use fire to cook because there's a chance it could burn your house down".

Well, jurisdictions are starting to ban gas stoves, so indoor cooking with fire might eventually become a thing of the past.

Generally hazmat labeling is only required when there is a clear and present harm possible. So far, the only argument presented is “theoretically someone could abuse the this” which is very different than “if you’re exposed to this, you will get cancer.”

If a theoretical argument that the masses can’t be trusted due to the actions of a few appeals to you, then I imagine you also support banning encryption to prevent terrorism.

The technology never should have existed in the first place. Adding to it is still adding additional harm.

Can't you take a moment and appreciate the great technical achievement before you ?