Generalizing individual experiences to general prognostications sells well as a genre, especially in software. This writing pattern was epitomized by OOP thought leaders in the 90s/00s, but was also present in the process evangelists who followed and the more iconoclastic folks like Dijkstra who preceded. However, actually following such general advice without taking into account the author's context can easily end in tears.
Ideally, these sorts of pieces should be written up as case studies rather than as general prognostications. With the context that a case study provides, these pieces can teach us a lot about how to engineer systems. Without that context, the reader is left to blindly follow advice that may not be relevant to their context. Or, best case, reverse engineer the context.
The fact that refitting once a day improves real world performance actually makes me think that the problem/data they work on is highly non-stationary, not that the model is memorizing. If it was purely memorizing then the model would perform poorly on all non-training data, and would not work for even one day.
To your point about large datasets - the large datasets are what allow learning to take place. With the most common forms of models we have now, they will memorize when they only have a few examples, and only “learn” when the training data is large enough. There is work to improve learning from a handful of examples, but in many of these cases they require a model that was already trained on the domain in question, and then are specialized to a specific use-case.
GPT-2 was for a time the best lossless text compressor (lossless compression is just memorization) - https://bellard.org/libnc/gpt2tc.html
I'm not convinced it actually pays, though.
Ideally, we would simply never stop training.
> It puzzles me when people say that small companies can’t retrain every day because they don’t have FAANG-style budgets. It costs a few dollars, at best, to retrain many xgboost or scikit-learn models.
if re-training every-day actually gives you a significant business material benefit (excepting cases where you specifically want to significantly over-weight recent data - phone keyboard autocomplete predictions as an example), you likely don't have a model that is actually picking signal from noise or generalizing on the data. This is memorization, not learning. This is essentially how you get expensive ML disasters.