Sorry this actually sounds like a real use case. What was the classification? (I tried google “sidebar archive”). I assume somehow you visited 20,000 web pages and it classified the text on the page? How was that achieved ? You ran a local llama?

We had ChatGPT look at 200.000 products, and make a navigation structure in 3 tiers based on the attributes of each product. The validation took 2% of the time it would have taken to manually create the hierarchy ourselves.

I think that even the simple LLM's are very well suited for classification-tasks, where very little prompting is needed.

Sorry to harp on.

So you had a list of products (what sort - I am thinking like widgets from a wholesaler and you want to have a three tier menu for an e-commerce site?)

I am guessing each product has a description - like from Amazon, and chatgpt read the description and said “aha this is a Television/LCD/50inch or Underwear/flimsy/bra

I assume you sent in 200,000 different queries - but how did you get it to return three tiers? (Maybe I need to read one of those “become a ChatGPt expert” blogs

I'm not this person; but, I've been working on LLMs pretty aggressively for the last 6ish months and I have some ideas of how this __could__ be done.

You could plainly ask the LLM something like this as the query goes on:

"Please provide 3 categories that this product could exist under, with increasing specificity in the following format:

  {
     "broad category": "a broad category that would encompass this product, as well as others, for example 'televisions' for a 50" OLED LG with Roku integration",
     "category": "a narrower category that describes this product more aggressively, for example 'Smart Televisions'",
     "narrow category": "an even narrower category that describes this product and its direct competitors, for example OLED Smart televisions"
  }

A next question you'll have pretty quick is, "Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category?" And that's a good and valid question, so you then have another query that takes the categories that have been provided to you and asks for synonyms, alternative spellings, etc, such as:

"Given a product categorization of a specific level of specificity, please provide a list of words and phrases that mean the same thing".

In OpenAI's backend - and many of them, I think, you can have the api run the query multiple times and get back multiple answers. enumerate over those answers, build the graph, and you can have all that data in an easy to read and follow format!

It might not be perfect, but it should be pretty good!

> Well, what if sometimes it returns 'Smart televisions' and other times it returns 'Smart TVs', won't that result in multiple of the same category

Text similarity works well in this case. You can just use cosine similarity and merge ones that are very close or ask GPT to compare for those on the edge

It sounds like a real use case, but possibly quite overkill to use an LLM.

Unless you need to have some "reasoning" to classify the documents correctly, a much more lightweight BERT-like model (RoBERTa or DistilBERT) will perform on par in accuracy while being a lot faster.

"while being a lot faster", yes; but something that LLMs do that those other tools don't is being hilariously approachable.

LLMs can operate as a very, very *very* approachable natural language processing model without needing to know all the gritty details of NLP.

I classified ~1000 GBA game roms files by using their file names to put each in a category folder. It worked like 90% correctly. Used GPT 3.5 and therefore it didn't adhere to my provided list of categories but they were mostly not wrong otherwise.

https://gist.github.com/SMUsamaShah/20f24e80cfe962d26af5315e...

> Every one I've spot checked right now has been correct, and I might write another checker to scan the results just in case.

If you already have the answers to verify the LLM output against why not just use those to begin with?

Not GP, but I would imagine "another checker to scan the results" would be another NN classifier.

Thinking being that you'd compare outputs of the two, and under assumption of the results being statistically independent from each other and of similar quality, say 1% difference between the two in said comparison, would suggest ~ 0.5% error rate from "ground truth".

Maybe their problem is using LLM to solve f:X→Y, where the validation, V:{X,Y}→{0,1}, is trivial to compute?

The problem isn't that it's not useful for self driven tasks like that, it's that you can't really integrate that into a product that does task X because when someone buys a system to do task X, they want it to be more reliable than 80%.

Stick a slick UI that lets the end user quickly fix up just the bits it got wrong and flip through documents quickly and 80% correct can still be a massive timesaver.

I think that can kind of work for B2C things, but is much less likely to do so for B2B. Just as an example, I work on industrial maintenance software, and customers expect us to catch problems with their machinery 100% of the time, and in time to prevent it. Sometimes faults start and progress to failure faster than they're able to send data to us, but they still are upset that we didn't catch it.

It doesn't matter whether that's reasonable or not, there are a lot of people who expect software systems to be totally reliable at what they do, and don't want to accept less.

We're thinking about adding AI to the product and that's the path I'd like to take. View AI as an intern who can mistakes, and provide a UI where the user can review what the AI is planning to do.

Except that people hate monitoring an intern all day, regardless of whether it is a human or a machine.

I think this is going to be a heavy lift, and one of the reasons I think a chat bot is not the right UX. Every time someone says “all you need to get to do to get ChatGPT working is provide it explicitly requirements and iterate”, and for a lot of coding tasks it’s much easier to just hack on code for a while, then be a manager to a 80% right intern.