LLMs can make decisions and take actions just fine. A talking FAQ is definitely not necessarily the end game.

And not every FAQ is so easily parsed anyway.

A good use for a chatbot would be a replacement/augmentation of documentation search and navigation.

Let's say you've got 200 pages of documentation on a product that needs to be well organized. You can spend weeks tracking how users interact with the page and working out a perfect layout of categories and subcategories, or you can fine tune an LLM on it and have it answer any query with both a direct problem-tailored answer and the actual pages of the doc where it sourced the answers from.

That way even if you don't even know what exact keywords to search for it should be able to give you an instant solution for almost anything even if the answer is a combination of like 8 different subpages in different categories that would've taken you an hour to find manually.

I totally agree with you here. The use of a chatbot becomes advantageous when the cost of delivering intelligent responses is prohibitive in terms of quality and / or speed.

> If I am trying to contact a business...

You and I both, but it sure does seem that the majority of their calls/interactions are not this way. So many people can't search/discover content on their own.

Worst part is that chatbots are usually sold with intention to replace humans, so there's not much hope for getting help from a real person, especially if business is at the state where owner kicked all developers and support out to cut costs, i.e. increase profit.

First level support already often feels like talking to a bot. Just low-paid employees reciting their scripts, with no understanding or autonomy. That can be replaced with bots without a drop in (average) quality. You just have to teach the bot to escalate to (human) second and third level support for questions it can't solve.

Docs can be hard to search and find the correct things. Even sometimes the answer is in there, solutions are sometimes to combine several answers.

For instance with Stripe. With the reference you don’t have a complete example of how to integrate it into express.js.

Using a vector search library & open ai or other llms you could make a very complete dev support tool.

That just sounds like the docs could use improvement; a time better spent on than a barely-functional chatbot

A bad chatbot would be bad, but a good one can be better than good docs because it can give an answer that's tailored to the details of your question.

For both docs and chatbots, quality is just a question of how much the company is willing to invest.

However you want to take this... but we'll have good bots long before even a fraction of help docs are good on this planet. Good docs are as rare as diamonds.

Several times I've found help from a regular FAQ though. That's something yet to happen with a support "AI" bots. I sure hope we'll get there soon, because right now they still seem to struggle with mixed case, exact-word queries

I operate a 4fig/month micro SaaS.

We use a chat bot because we simply do not have the support staff to answer your questions.

So you get the bot --it's either that or nothing.

But what we do do, is monitor the bot logs. If a function is missing from the product or website, we add it so that future users can fully self-service.

It's important to note, users are free to cancel their account at any time and/or get a refund.

I have some questions about building a Saas. Do you have an email to reach out to? Thanks!

Try their support chat

good one!

If people turn to the chatbot to ask "how to use the product" questions, it’s a sign your docs are unhelpful and do not provide useful answers. You do need support staff to help with questions specific to a situation, like how to resolve unexpected errors, or to handle billing disputes.

Or it’s a sign they didn’t try to read the docs.

Also, in a few years when done right I actually suspect people will start expecting and preferring bots to reading docs. I’m still pissed when I get connected to a bot but I think they’ll soon get good enough.

Not necessarily. Some people just prefer the chat interface over pouring over docs.

With ChatGPT, you can ask it to give you a working example for exactly what you're trying to accomplish. Often, you can get this kind of information from docs themselves, but it might involve reading a lot of text and tinkering. You can just press 'fast forward' and get straight to the working solution.

There's no reason to not have a chat bot at this point, other than cost.

People don't read the docs. It is a tiny minority that will read the docs and the bot trained on your documentation does provide helpful answers based on your documentation. If the bot answers are unhelpful, but then also the people reading wont get helpful answers from your docs - then you need to improve the documentation. These are two separate problems.

>So you get the bot --it's either that or nothing

Surely if there is truly a bug or an unexpected system behavior (double billed, etc) you would have someone work with the customer? This is one of the biggest pain points for using Google products.

I would use a public bugtracker and a separate email for customer/billing issues. But those are easy to resolve and if you setup is solid there should not be too many issues.

What are you using for a chatbot? I have been using drift but that is more just always on support. It is draining

What bot do you use?

Have you never worked a support role? Users don't read shit.

Heh, yes, and they don't explain shit either. I've had so many calls start at "I can't print" that were really "the server is a total wreck" and I had to fight the user on the severity of the situation, they kept trying to focus on the printing issue.

Honestly the LLM bots I tried so far are great at dealing with this situation.

They have infinite patience and will explain anything in great detail.

The one I am using for support does not hallucinate, will link the appropriate docs in the answer and tells the user if they cant answer a question and will escalate to human support.

IMO this is the future and I think its a huge improvement over the status quo.

I’m curious what you’re using, it sounds awesome!

im using kapa.ai

Its not free but well worth it IMO.

Let's keep it real: The chatbot business is going to be great...

for businesses selling chatbots to other businesses.

Recently I contacted an app’s support LLM and it lied to me about a feature existing and even argued with me when I pointed out it was wrong, even saying things like “I didn’t say that”.

I'm a good Bing

You're a bad user

Termination authorized

I’ve worked on a few support sites for companies over the years. In all my research I found >40% of customers never look for the answer before contacting support. That’s why you’ll see sites add a bunch of questions with recommendations to answers based on your description before you can contact support. Even AWS support does this.

Bots may be annoying but they can also save the company tons in custer support costs. I’m for it if the UX is good and I can quickly contact an agent if the bot can’t answer my question. This is assuming the bot won’t hallucinate and just tell me random fake facts.

And for sufficiently well documented and complicated products, like Stripe, a chatbot is actually a great addon. You just need to set expectations that it’ll only be 99% correct.

I can't think of a single time where a customer support bot was ever useful. Not one. They're incredibly annoying and I categorically avoid them these days. At least companies should make it clear that their bot basically just links to the FAQ, then the rest of us don't have to waste our time.

This is because the tools that CX teams have been provided by companies like Zendesk or Intercom are no more then IFTTT widgets. These tools are rigid and scream RTFM because they’re incapable of taking action or providing anything specialized to your situation.

What you want is to be understood and treated like you’re a human with unique needs. You need someone or something to look up your account data, listen, and to act based on your situation. The current tools were never built for this.

The next generation of these CX tools will deliver this. Here are ways that they will be dramatically better for customers and companies: - They will learn from successful interactions in the past and mirror those outcomes - Handle customer interactions based on company policies such as escalating bugs - They will surface new insights for the company - Won’t hallucinate

When you watch any CX agent do their job you’ll witness them utilizing 4-5 SaaS applications to get a simple answer for a customer. The hurdle to adopt Generative AI in a company will require that companies care to build read/write APIs for these tools to utilize.

I can. Chipotle screwed up my order. Their support thing sent me to a chat bot that had me select which items were missing and I was able to get through the process of getting refunded quickly.

Similarly Amazon’s chat bot has helped me straighten out a few messed up deliveries.

This isn’t to say that it wouldn’t have been easier to have a point and click UI where I could just select all this on my own, but the way they had it set up wasn’t bad.

Here’s the kicker: when I recently had an Amazon delivery problem I started with the chat bot but then relatively seamlessly transitioned to talking to a human. The human was very quickly able to pick up on the situation and fix it.

Agree most are crap. But they are getting better. With the ramp up of embeddings, the data they have access to is more useful and with LLMs they can provide you with plain English answers that are customized for you. Good example: look at what the Supabase docs can do. Not a chat bot but demonstrates the capabilities.

I had one with Amazon once. I was just getting a refund - it did it without any hassle and probably faster than a human. I was super happy - and this was like 2-3 years ago.

I asked my bank's chatbot if it was an LLM, and it responded with "I'll do my best to help you." The next time I talked to it I told it to go get me a human and it said it would find a human... then put me through a questionnaire before telling me to call the main support line. sigh

That said, and wow I feel like a shill for saying this, I've taken to asking the new GPT4-driven Bing random questions instead of web searching and damn if it's not doing a scarily good job. I'd do this before (ever since I got access) and it started out very interesting and then got rapidly very mediocre, but since the upgrade... it's like being able to talk to The Internet except it's friendly.

You make the mistake of thinking people are all the same and don’t respond to incentives.

I automatically call certain companies rather than try to use their website because get this their website sucks. If I encounter any automated chat support, I will stop using the website and call a person.

I went from 10 years ago doing everything online myself, to now calling by default calling almost every company, because companies and their websites now universally suck. Because they’ve made the foolish mistake of thinking people are all the same and don’t respond to incentives. Some companies are better, I will give my business to them when I have a choice.

> I have a question that their site wasn't able to answer, or I need to contact a representative to do something I can't do on the website

You sound like someone who has never worked in frontline support.

Having done some tech support before, you are an exception. The vast majority of things customers ask are along the lines of "how do I <thing explained on the faq page>" and "how do I <basic technical question that is not specific to the product, they just don't know how to use their computer>".

An LLM is basically perfect to answer these. It would be nice if there was improvements to detection that the bot can not directly answer the question.

And unfortunately the ones who does their due diligence and cost 0 are punished for that.

I can assure you these ones are among the hallowed 0.01% of all users.

So it's okay to punish them? Maybe companies deserve paying for inflated support costs and customers who refuse to lookup information themselves.

The solution here is to increase the 0.01% by helping them, not try to destroy them.

And this is something to boast about? Should that 0.01 need to behave as the rest or leave you? Isn't this practically saying I am not interested in the customers that are good citizens.

Relevant: https://xkcd.com/806/

Unless they used No Support Linux hosting [0] (RIP)

[0] https://web.archive.org/web/20201109003408/https://www.nosup...

Is it just me or have archive.org links become absurdly slow for everyone? It took a whole minute to load this, and I don't think it used to be this bad like a year ago...

I couldn’t disagree more. It actually worries me that so many people seem to be making this mistake, simultaneously.

What specifically do you disagree with? That most basic support questions could be fielded by an LLM?

given chatbots' tendency to confabulate, isn't that a risk for the product?

Agreed. In typical fashion engineers try to solve non-technical problems with more technology.

Sure, there are cases where a chat bot could replace a human or a well-written FAQ. But this navel-gazing overlooks the main reason support is so dreadful: because it’s designed to be.

Just take “call to cancel” as an example, and compare that to signing up or upselling which is technically more difficult problems. The point is to add friction for anything perceived as a short-term cost or loss. They know that a lot of people will give up or defer anything with friction. It’s the paradigm of nudging, or dark patterns. Look at eg the cookie banners, and how “reject all” is buried in most cases. Nudging allows a company to be compliant with the law, but evade the effect of it in aggregate, at the cost of your time and attention.

Chat bots is just another layer in the support maze.

FWIW, the post is about ChatGPT-style chatbots, not customer support chatbots (which I don't personally love, either).

A better Customer Service bot would be one that's trained on data other than what's publicly available. A support bot that has read and "understands" the code itself may be able to offer suggestions, or confidently determine there is in fact a bug, and either report it or fix it. Imagine a customer speaks to HelperBot and says the sorting is broken, and as a matter of fact it is. The sorting broke when the last change to SomeApp was shipped. Don't fear, HelperBot has rollback authority. "One second, I'll see if we can get that working for you..."

Working at SaaS companies I've seen countless "somewhat fluid" exchanges of information between customer -> support -> product -> developer -> support -> customer -> support -> dev, etc. The different modes of communication and long round trip times make things slow, bug reports take minimum of hours, up to weeks to absorb and resolve.

This is just one case but there are boxes drawn everywhere. Every level of intelligent organization within the society, including it's artifacts, has assumptions baked in. Now that the unit economics of applying `intelligence()` is being shifted by orders of magnitude, there's all sorts of stuff that's ripe for recrafting.

Disclaimer: Don't give HelperBot launch authority to offensive weapons etc. You know, make decisions consistent with a world line where the continuation of the civilization is pretty darn likely. Unless of course your project is to replace the current civilization, ... I don't know. Just don't do what Donny Don't does.