Glad you like the post.

I strongly agree with all of your points, especially custom loss functions can be a great tool. If the problem you are trying to solve has some grounding in e.g. physics you can even go a step further and let the model itself mirror the physical equations.

It's like you say, of course these big models are really cool, but I feel like most of the popular machine learning online courses are too narrowly focused on them and many people discard useful techniques if they are not popular in kaggle competitions.

I fully agree with your perspective, and I think there's a lot of cargo-culting in that area that explains your observations. Sure, if you're a FAANG collecting massive amounts of data comes almost for free, and it makes sense to find a way to properly utilize that data. But for many startups collecting that amount of data doesn't make sense (either because they don't have a lot of users yet or because their domain isn't high-freq digital activity or both), and in those cases it can be more worthwhile figuring out what to do with the little data they have than shoe-horning their way into Big Data (TM).

There's much more money to be had convincing startups to use big data infrastructure and train huge nets on GPU clusters etc. than convincing them to iterate on small ad-hoc models developed in-house. Not saying that big data or models are necessarily wrong for small startups, just that they don't have to be the default.

One question (please take in account that I am no more than a dabbler in this field) - you conclude your piece with ... the nature of the problem slowly changes over time and prediction quality deteriorates.

Isn't this a problem anyway, even with models based on much larger datasets?

You are right, data/concept drift affects both approaches.

What I meant to say was that retraining your model might not fix that if you baked strong assumptions into it. I edited the blog post to make it clearer.

With small datasets you need to know how the concept drifts well enough to model it or model the failure modes. That's half the game in my opinion.

I’ve found constraints in the loss functions are key to finding the correct solution space. With small amount of training data and SGD you can get a lot of mathematically “correct” answers, but a well informed constraint based on your problem space can eliminate 99.9% of the mathematically correct but practically incorrect answers.

Another approach is to use Bayesian, and state some meaningful priors. This is implicit regularization (constraining) since the priors are "pulling" back the parameter values based on the data (likelihood), and in the end you get some kind of compromise between the two (posterior). One example I have is when estimating traffic flow, and I put a Poisson (or Half-Normal - works ok as well) distribution as my prior, with some gut-feeling/logically-reasonable parameters.

Could you elaborate a bit more on how you do this, or point to resources on it? Do you mean putting constraints on possible parameter values, or just on the loss function values itself? Thanks.

I have never dabbled with constraining the parameter values themselves. Ive mostly put the constraints into the loss function. This works well when working with regression, a super simple constraint that adds penalty when the regression goes outside of the possible solution space has been extremely helpful in our work. Heuristically, I think it’s more useful during the first few iterations to find the correct local minima. If you were already finding the “correct” local minima this might be less important, but if you’ve ever dealt with convolutional artifacts (boxes, lines, edge effects) in your predictions, well informed constraints tend to help avoid these, as they are a symptom of being in an incorrect local minima.

Makes sense, and designing proper constraints is an art on its own.

Out of curiosity, did you apply the constraint as a term in the loss or did you use constrained optimization?

I’ve only ever played with constraints in the loss at this point. Would love more time to also deal with constraining the optimization. Do you have any examples that come to mind that worked well when doing this? Thank you for your suggestion!

Thanks for your reply!

I've only seen (and used) this for linear / convex models and constraints, so that's actually linear / convex programming (highly recommend the cvxpy library [0]). I was curious if you've integrated that into DL models.

Here's a fun example I did for a uni project: say you have a small recipe dataset, where for each recipe you have the ingredient list ("3 tbsp sugar", ".5 kg flour" etc.) and macronutrients (carbs, proteins, fat), and you want to learn to predict the nutritional content given a list of ingredients.

With a bit of text manipulation you can split each ingredient text to "amount", "measurement unit" and "ingredient type". Then you can decompose the nutritional values into nutritional density of ingredients and the conversion of measurement unit -> actual mass for each ingredient type. Then you can introduce constraints in the form of known conversions between units, e.g. 1 cup = 16 x tbsp, known nutritional densities of some simple ingredients, and known unit conversions for specific ingredients.

It worked better than simple regression (that didn't take into account unit conversions) and a simple MLP, though not sure how it would compare if you actually tried to finetune a language model to the task. The overall predictions weren't too accurate, but for common ingredients it actually gave very close nutritional density values (without being constrained), which was cool.

Edit: another constraint I initially forgot about is that each gram of an ingredient cannot contain more than a gram of macronutrients (it can contain less because water), and of course nonnegative nutritional density.

[0] https://www.cvxpy.org/

Thank you too for this information and for the resource, I'm looking at it now it seems very interesting.

The breakdown you give above seems to me to be more akin to something DL types tend to call 'feature engineering'. I also have a fun example, in this case it would be identifying land cover from satellite imagery. You can obviously just feed the raw reflectance values (RGB etc.) into a DL model to create a semantic segmentation of classes. However, it's been well established in literature at this point that that is not the most effective way to create a classification. This is similar to my previous comment, where there are lots of solutions that can be found through SGD based on these raw values.

There's a lot of traditional satellite imagery analysis algorithms that are based on very simple 'band-ratios', i.e. NDVI (normalized difference vegetation index) is calculated by Near Infared - Red / Near Infared + Red. This index will visually highlight areas of vegetation that was extremely useful in human sight-based analysis to identify vegetative areas. Now, you'd expect a deep learning model that takes in all the bands to have this information already, it has the NIR band and the Red band. However, explicitly doing the NDVI calculation and using it as an input feature leads to increased accuracies for classification. The exact reasons for this are unknown, but I think you touched on some of this above. With machine learning, and DL even moreso, sometimes it's necessary to hand-hold the optimization to optimize for exactly what you want. It helps 'explainability' and oftentimes helps accuracy, at the cost of some preprocessing.

That's cool, I didn't know that, thanks!