And you can now get that world knowledge out of llms and generate synthetic data to train your svm and fine tune on your own dataset!

I’ve hand-labeled so many freaking datasets over the years for supervised learning tasks, if only to build out a PoC. Would 100% hand this off to an LLM and just fine tune the results. I definitely do not want an LLM in the loop at inference time for anything I could build a simple classifier for.

Llm labelling is so incredibly useful, as is transforming labelled data to other valid but different forms (like where you might train on sentences but also randomly replace days of the week with others).

You can also do stuff like setup active learning but have the labeller be an llm as well to make things more efficient.

Definitely worth it for PoC level things if not full ones.

LLMs, and other transformer models before them like BERT for example, have the benefit of "transfer learning."

If you want to make a ticket classification model, you don't need to train across many customers doing ticket classification, who might not want you to train on their data and whose data is impossible to obtain unless you already are a large company. Instead, you can train on a completely different task which forces the model to learn the syntactic and semantic structure of the English language — say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English — and then fine-tune that model on your specific task with much less data.

Because the initial task is so general, you can have one company do the expensive pretraining, share the base model, and then have everybody else just do the finetuning on their (small) proprietary dataset.

LLMs and models like Jev go one step further, letting you use them as classifiers with no training at all, with nothing but rules in the prompt. And with a few tricks, you can indeed use them to get pretty high-quality labels, which you can then distill into a fine-tuning dataset for a much smaller model.

  > say predicting missing words in Wikipedia articles, which you can't do successfully without a good understanding of English

This is word2vec model, one of - predict a word from context. The other one is to predict surrounding words (context) from a word. It did a good job when I used it, words' classes were impeccable for most practical purposes.

Word2vec is a two layer perceptron.

¿Por qué no los dos? Big skill issue, you can even route known problems to a trained in-house simple classifier and low-confidence problems to Jev-like "zero shot classifiers" or LLMs.

I feel like people in the field forgot everything.

It will only get worse. Non-tech people will be (or are) very suspicious of anything trained in house. The bar to spend time/money on anything domain specific will only get higher.

Agree to all of this but I would make a minor correction: the interpretability is true under certain conditions, e.g., you use something like n-grams and train an SVM with a linear kernel. You could change the representation back then too, e.g., LDA (topic models), Latent Semantic Indexing, and of course, you could change the SVMs kernel as well, e.g., use a sequence kernel [1]. Some of these were much less interpretable. But the common pipeline back then was indeed n-grams + Linear SVM. Today it is easy to modernize the pipeline by just changing the representation step to use an embedder like ModernBERT. I have a "one page" classifier if you want to try it out - [2].

[1] I went down the rabbit hole of using sequence kernels quite a bit back then, see https://arxiv.org/abs/1910.13800, but the timing was terrible. RNNs and variants like LSTMs were beginning to become popular again, and everyone (including me) switched.

[2] https://pocket-text-lab.abhishek-ghose-82.workers.dev/