A simple bag-of-words classifier requires some feature engineering, parameter tuning, continuous retraining for generalization, and actual thinking on the engineering end. Simply tossing a small amount of data at DeBERTa v3 (for classification/regression tasks) will give the same or better results 99% of the time, with a slight increase in inference cost and significantly less human effort.
You might have some domain where the new models work for you and I'd love to hear you talk about it.
I see a lot of papers go by in arXiv and also blog postings by data science people and would say that the behavior of a classifier can be limited by many things. For the most part bag-of-word classifiers do very well for classifying topics because topics involve very different vocabulary. They do not do so well at sentiment analysis where you have to know "not good" = "bad".
I worked at one place that had a CNN classifier that could classify random snippets as "address", "full name", etc. but it wasn't able to learn how to calculate credit card checksums.
For some problems 90% accuracy is very bad (e.g. predict some event that happens 1 in 10 times like a headline getting a few comments on HN -- a fancy classifier could probably do better than my simple classifier it is not going put up a dramatically better AUC because of the fuzziness of the problem) Even crisper concepts get controversial around around 1 time in 20.
My simple classifier is fast to learn I like articles about classifiers and don't like articles about theoretical CS but it struggles to tell I like the NFL and hate the Premier League, a fancy classifier could do better, and I will give one a chance because i have the data to do it with.
With the simple classifier it is simple to do cross-validation, parametric tuning and such, but usually people publishing results on deep models do not publish error bars, do not understand how the quality of the model varies from run-to-run, etc. Even if you ask ChatGPT to do it you will need to supply a large number of test cases to prove it gets the result.