model drifting is something a lot of folks do face after 5th/6th turn as per my understanding and it usually the median, how did you tackle it if you have yet?
also yea, thats why we went for a per customer taxonomy than a general one, yeilded better results + easier to improve upon.
Regarding my experience, I have done a fair amount of work in the contact center space with long calls. I used statistical Bayesian approaches which I found to be much more resilient especially on long documents than embeddings/transformers. It also provided a joint modeling foundation for classification with much lower label requirements than BERT or traditional ML.
damn, i read bayesian in statistics like years ago, never thought itll come back this way
How is it easier to sign up and manage a different service, implement a different API, etc.
And from the company side the fatal flaw is that these types of tools rely upon 1% of their users having huge spend. Nobody is going to be a huge spender here because it's easier to hand roll than navigate procurement on this (not to mention impossible to justify the spend, additional security/privacy risk, etc.)
It feels approximately impossible for this company to have large accounts.
we're keeping it useful every week, finding out insights that the teams can extract value out of, work with them to understand users better.
the procurement what we've seen is v similar to how one would have for any analytics product? and we're selling this to companies when/once it becomes someone's job to do this
Realistically do you also need to have this live with a fancy? i.e. a custom solution maybe even Jupiter notebooks initially might be sufficient. It's not like 100k messages is a very large dataset. It's not trivial to make a generic solution that fits every use cases (besides of basic customer chatbots) to get actual value for more agentic products.
we're also targeting conversation first use cases and for them this serves as their everything custoemrs. we obv do not sell the fancy part, idts that sells anymore lol. lot of our queries come from our slack app/mcp.
At the scale of a few hundred to a few thousand documents, especially short documents, there are a few out of the box methods that can yield reasonable results, whether it be embedding clustering or leveraging LLMs for tagging.
However as your (1) datasets gets larger (2) documents expand from tweets and text messages to 30+ minute conversations and (3) you build downstream analytics on top of the learned semantic units, you really start to feel the limitations of LLMs and embedding for reliable annotation. That doesn't even get into the nuances associated with taxonomy management, seasonality, and model drift.
TLDR; this problem solved effectively has a lot of value and is a lot harder than it seems.