Launch HN: Retell AI (YC W24) – Conversational Speech API for Your LLM
With the advent of LLMs and recent breakthroughs in speech synthesis, conversational voice AI has just gotten good enough to create really exciting use cases. However, developers often underestimate what's required to build a good and natural-sounding conversational voice AI. Many simply stitch together ASR (speech-to-text), an LLM, and TTS (text-to-speech), and expect to get a great experience. It turns out it's not that simple.
There's more going on in conversation than we consciously realize: things like knowing when to speak and when to listen, handling interruptions, 0-200 ms latency and backchanneling phrases (e.g., "yeah", "uh huh") to signal that they are listening. These are natural for humans, but hard for AI to get right. Developers spend hundreds of hours on the AI conversation experience but end up with poor experiences like 4-5s long latencies, inappropriate cutoffs, speaking over each other, etc.
So, we built Retell AI. We have followed the overall paradigm of having speech-to-text, LLM, and text-to-speech components, but have added additional conversation models in between to orchestrate the conversation while allowing maximum configurability for the developers in each step. You can think of our models as adding a “domain expert” layer for the dynamics of conversation itself.
Retell is designed for you to bring your own LLM into our pipeline. Currently, we can achieve 800ms end-to-end latency, handle interruptions, speech isolation, with tons of customization options (e.g., speaking rate, voice temperature, add ambient sound). We created a guest account for HN, so you can try our playground with a 10-min free trial without login: https://beta.retellai.com/dashboard/hn (Playground tutorial: https://docs.retellai.com/guide/dashboard). Our product is usage-based and the price is $0.1-0.17/min.
Our main product is a developer-facing API, but you can try it without writing code (e.g. create agents, connect to a phone number) via our dashboard. If you want to test it in production, feel free to also self-serve with our API documentation. One of our customers just launched, and you can view their demo: https://www.loom.com/share/64f09a53bf6d4b3799e5ebd08b23fec4?...
We are thrilled to see what our users are building with our API, and we’re excited to show our product to the community and look forward to your feedback!
Retell focusing on all of the other weird/unpredictable aspects of human conversation sounds super interesting and the demo's incredible.
Things are moving so fast, wow.
"What time works"
"Morning on tuesday would be best, but I can also do afternoon"
"I'm sorry, I didn't catch what time in the afternoon you wanted"
"No, I said the morning"
"I'm having a hard time hearing you. What time in the morning did you want?"
"10am"
And from there things were fine. It seemed very rigid on picking a time and didn't suggest times when I laid out a range.
oh no
Amazing product, looking forward to working with it.
What is worse, that no one is available to listen to you when you're suicidal, or that you lack so much value that only a machine would talk to you. I'm sure some people would have an extremely poor reaction to that.
I'm going to move all of this to an offtopic stub and collapse it.
We tell founders to make sure this doesn't happen (see https://news.ycombinator.com/yli.html) but I probably need to make the message louder. Not everyone understands that the culture of HN doesn't work this way.
Can you please ensure, going forward, you have the universal "truth" as it were, to have your system always identify if its AI when "prompted" (irrespective of what app/dev has built - your API should ensure that if "safeword" is used it shall reveal its AI)
--
"trust me, if you ask them if they are a cop, they legally have to tell you they are a cop" (court rules its legal for cops to lie to you) etc....
(it should be like those tones on a hold-call, to remind you that youre still on hold... but instead its a constant reminder that this bitch is AI) -- there should be some Root-level escape word to require any use of this tool to contact a Human. That word used to be "operator" MANY times, but still...
Maybe if a conversation with an elderly Human goes on with too many "huh? I cant hear you" or "i dont understand, can you repeat that" questions, your AI knows its talking to a non-tech Human, and it should re-MIND the Human that youre just an AI. (meaning no sympathy, emotion, it will not stop until you are dead) etc...
Guardrails, motherfucker, Do you speak it!"
a dev builds an app it | to your API and you spit it back out? - if so - ensure when you spit out whatever it defines itself to whomever is listening....
--
Plz explain the arch of how your system works? (or link me if I missed..)
----
Shortest and most importnat law ever written:
"an AI must identify itself as AI when asked by Humans."
0. Law of robotics.
------
@autsin
-
Cool - so im on an important call with [your customer] your system has an outage?
How is this handled? dropped call?
(I am not being cynical - im being someone who is allergic to post mortems.
----
EDIT: you need to stop using the term "user" in anything you market or describe. full stop.
the reason: in the case of your product, the USER is the motherhecker on the phone listening to anything your CUSTOMER is spewing at them VIA your API.
the USER is who is making IRL *>>>DECISIONS<<<* based on what they hear from your system.
Your CUSTOMER is from whom you receive money.
THEIR customer, is whom they get money to pay you.
The USER is the end-point Human. who doesnt even know you exist.
Our homepage https://www.retellai.com/ has a GIF on it that illustrates this point.
For outage handling: we strive to keep up 99.9 plus up time, and in the case of a dropped call, the agent would hang up if using phone, and might have different error handling in web depending on how customer handles it.
> we focus most of our energy on innovating the AI conversation experience, making it more magical day by day. We pride ourselves on wowing our customers when they experience our product themselves.
This is not useful; you already have testimonials to show what customers think.
Maybe convert that first FAQ point about differentiation into a table comparing you against the closest competitors. Since you talk about performance you should measure it. Use a standard benchmark if there is one for your field.
People call customer service because they don’t know what to do. It would be better for most customers to talk to a bot that they can catch making a mistake.
Recent example: https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-th...
For example - "hallucinations / LLMs confabulate": techniques like RAG can help - "Language is so smooth that you often need the expertise to catch them in it", fine-tuning and prompt engineering can help
This blog breaks it down well:
https://www.papercup.com/blog/realistic-synthetic-voices
P.S: I tried to fool the Dental Office demo trying to book on Sunday or outside of the slots it had indicated, and it did a better job than many humans would have :)
As a customer of a service like Retell (though of course not specific to Retell itself), how might one go about setting up rules to keep a phone conversation from going on for too long? At 17¢ per minute, a 6-minute call will cost just over $1, or about $10 per hour. Assuming the AI receptionist can take calls outside of business hours (which would be a nice thing for a business to offer), then such a malicious/time-wasting caller could start at closing time (5pm) and continue nonstop until opening time the next day (8am), with that 15 hour span costing the business $150 for billable AI time. If the receptionist is available on weekends (from Friday at 5pm until Monday at 8am), that's a 63-hour stretch of time, or $630. And if the phone system can handle 10 calls in parallel, the dentist could come in Monday morning to an AI receptionist bill of over $6,300 for a single weekend (63 hours × $10 per hour × 10 lines).
This is in no way a reflection on Retell (I think the service is compelling and the usage-based pricing is fair, and with that being the only cost, it's approachable and easy for people to try out). The problem of when to end a call is one I hadn't considered until now. Of course you could waste the time of a human receptionist who is being paid an hourly wage by the business, but that receptionist is going to hang up on you when it becomes clear you're just wasting their time. But an AI bot may not know when to hang up, or may be prevented from doing so by its programming if the human (or recording) on the other end of the line is asking it not to hang up. You could say it shouldn't ever take more than five minutes to book a dentist appointment, but what if the person has questions about the available dental procedures, or what if it's a person who can't hear well or a non-native speaker who has trouble understanding and needs the receptionist to repeat certain things? A human can handle that easily, but it seems difficult to program limits like this in a phone system.
It'll be like that San Junipero episode of Black Mirror - immortality in a dark way.
If your looking for phone flavour, also very good. Curious which is 'better'
However, I agree with others. What I don't want to do is talk to LLMs for customer support. For that, I always want to talk to a real person. It would be infuriating for an LLM to pick up when calling a business.
Any plans for something like that? It'd be really interesting to have prospects what with an AI version of me to answer some basic questions.
For example, take Amazon Connect pricing:
There is an Amazon Connect service usage charge, based on end-customer call duration. At $0.018 per minute * 7 minutes = $0.126
There is an inbound call per minute charge for German DID numbers. At $0.0040 per minute * 7 minutes = $0.0280
If I were to bring retell into the loop, I'm changing my self-service per minute cost from .018 + .004 = .0184 per minute to .1184 per minute at the cheapest setting. And from that, I don't have a clear case for the impact on deflection, and because of that I don't have clear ROI.
This isn't saying that the product isn't great -- but at the current price I struggle to see how anyone at scale can use it without eating the double whammy of LLM + Retell costs.
Guessing you guys have found a use case with higher margins than ours which'll explain the price. Great work. Hope we can afford this one day.
Well I'll see your demo, and raise you another demo
https://x.com/sean_moriarity/status/1760435005119934862
Except that demo was done by a single person, with a day job and no cap table.
Playing around with the Nero demo, it also feels pretty... broken? I never actually managed to make a booking with Nero; it seemed to often not be able to tell I was done speaking, and would just hang after a couple back-and-forths. Still impressive for a one-man demo built quickly, but they don't feel in the same league.
I do wish Retell's pricing was cheaper, though; $6/hr is pretty much the cost of a call center employee in India, and LLMs still perform below the average human on most things. That being said, I imagine the cost of Retell will come down as the tech advances, and LLMs will also improve over time.
The unnatural voices happen I think because for some reason the way I stream audio to the browser breaks up the audio from ElevenLabs, so you get almost this “start/stop” sound that makes it sound worse than usual.
There’s a lot of minor details to get right, but it’s a fun problem to try to solve. I wish Retell the best :)
Python demo: https://github.com/adam-team/python-backend-demo
With that out of the way, it's really interesting. The challenge I suppose is how narrow / wide the API made by the developer is. A narrow case, like "booking a dentist appointment", might feel like a step down compared to an online form, and most likely would fail short of satisfying someone calling, because if they're calling they had some deeper needs.
On the other hand, a wide API, like if you gave access to pricing information of the dental practice, info about the doctors, a way to reach an actual human being, health advice and post op advice, generate documents from past appointments etc, and you have a higher chance of hallucination, misclassification of what the user wants etc.
I'm still not sure if the best approach is just to leave the AI to deal with that, or to represent the user intent etc. with some sort of State Machine. Maybe a case by case etc. But the more you add in that logic, the slower the machine is.
I think the main issue right now for personal use would be cost (and I'm guessing STT / TTS are the most expensive parts..)
https://news.ycombinator.com/item?id=39467413
Would it be possible to integrate Retell with a RAG pipeline?
This demo really shines light on how you could "talk" to place your order in a comparatively fast amount of time, similar to speaking with a human. Yes it can't handle all the million corner cases, and it will get it wrong a bunch of the time, but at least you can verify your order on a screen before tapping your phone to pay.