A human will be more likely to say "I don't know" or pass you along, rather than outright lie.
However I would not expect an airline customer support to make up a completely fictional flight that has never existed. Maybe they could confuse flights or read a number wrong, but making one up?
Case in point, I asked my bank if they had any FX conversion fees or markup. Guy said no. I asked if there was any markup on the spread. Said no. Guess what? They absolutely mark up that spread. Their exchange rates are terrible. Just because there isn't a line-item with a fee listed doesn't mean there isn't a hidden fee in there. He's either incompetent or a liar.
90% of my job was undoing and compensating passengers for the incorrect information either the phone agent or gate agent gave them. The other 10% was dealing with workarounds to technical issues in our booking software.
I recently asked some LLMs "How many gallons in a mile?" and got some very verbose answers, which turned into feats of short story short stories when I refined to "How many gallons of milk in a mile?"
If part of the training was to only use knowledge sourced from a vector db and that it is allowed to use its trained knowledge only for grammar rules, phrasing or rewriting information then I think it would do a lot better.
Doesn't seem like many models are trained on prompts like "Question Q"->"[no data] I'm sorry but I don't know that" = accepted during training.
This would help immensely for not just for chatbots but for personal use too. I don't want my LLM assistant to invent a trip to Mars when I ask it "what do I have to do today" and my calendar happens to be empty.
And especially for something where it’s just pulling data from an internal system. There is absolutely no reason to invent made up information and saying “well humans do it all the time” is just an excuse.
On the phone with a customer service rep, I might understand a little wishy washy answer, slip of the tongue or slightly inaccurate statement. I've never really had a rep lie to me, usually its just I don't knows & them escalation as needed.
There is something about the written word from a company that makes it feel more like "binding statement".
They only need to increase the lawsuit/settlement amount by less than the amount the companies saves by automation.
If a company representative told me in writing (perhaps via email support) that I could claim a refund retroactively, and that turned out to not be their policy, I would still expect the company to honor what I was told in the email.
Phone calls are difficult more because there is no record of what was said. But if I had a recording of the phone call... I'm not actually sure what I would expect to happen. It's just so socially unusual to record phone calls.
Is it? I can not remember the last time I called some business where I did not get a “this call may be monitored or recorded for quality and training purposes…”. whatever perceived social hangups the company had they got over them and you don’t even need to ask in a 2PC jurisdiction, it’s already taken care of, just record the call.
The bar LLMs have to clear to beat the average front line support operations isn't that high, as your own experience shows. And compared to a large force of badly paid humans with high turnover, LLMs are pretty consistent and easy to train to an adequate level.
They won't beat great costumer support agents, but most companies don't have many of those