> The advantage of physical systems is that it's straightforward to collect data from natural systems.

For a lot of physical systems it might be "straightforward" but expensive. Testing new building designs for energy efficiency is just a matter of building a new test building, why not just build 1000? Testing new car designs to see if they reach the safety qualifications is just a matter of crashing a few of your prototypes, why not crash 1000? It's just a matter of getting more telescope time. It's just a matter of getting more robots to try more materials or proteins. Yes you can do theoretically do it, it's just cost prohibitive to get Google-level datasets for many scientific and engineering problems. Of course that's not true for all domains (high-throughput sequencing for bioinformatics is the prime example of something that became cheaply data-rich for standard machine learning), but there are many domains where getting another data point is always possible but just would cost another million.

yes, but it's still unclear that for data-poor physics problems (the joke in grad school is that you could graduate when you collected your second data point, since that was enough to draw a straight line, and we all know any interesting physical phenomenon is linear in the engineering regime, right?) that NNs really provide anything more than a bayesian prior that physics folks already knew how to incorporate.

High throughput sequeuncing didn't solve any problems for bioinformatics- in fact, the problem is that we have so much data we don't know how to process it to extract the actual value from the data.

The general principle can be referred to as "inductive bias": out of the space of all functions that can be represented by computational means, how does one restrict the search space to only those functions which are relevant for the data/system of study? Developing physical theories is nothing more than the process of whittling down what space of functions can represent the data most accurately, using analytical and scientific techniques to get there.

Bioinformaticists, with sufficiently accurate and detailed models of how genes translate to $BEHAVIOR_OR_OUTCOME, would need far fewer data to extract meaningful insights. So of course throughput hasn't solved the problem. That wasn't the point of the comment above.

I don't think there's any deep relationship between the functions we find that fit observed data well, and those functions being generally useful at making out of class predictions.

Really, I was making a comment about https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... which in my mind is still unresolved (that is: why do the math models we use for physical systems generalize so well, if we wouldn't expect them to do so?)

When you can model a system using physical or known principles the data points are mostly there to prove the model is valid. Also they usually aren't randomly chosen data points like in ML they are chosen via statistical theory.

The disadvantage of physical systems is the data is usually expensive. In my old field I know someone who had to make a decision on a data set of 5 samples and each sample costed more than a million USD.

I've made models of physical systems that involve kind of life or death stuff with less than 30 samples.

It's not for the faint of heart and if someone doesn't know what they are doing, the system they are modelling, or how to share their results correctly... Yikes!

can you share more about models of life and death stuff with less than 30 samples? (I work tangential to a regulated health industry)

I'd like to stay anonymous on the matter, but I've worked in various regulated industries. Now days I do none of this .

Uh, OK. because if you were doing life and death stuff with 30 samples, people probably died due to false negatives or positives in your model, in a way that would have been better handled with more samples. Sorry to hear it- it's one of the reasons health care is so shitty.

Not healthcare and no one died. My models have been deployed for two years without incident.

oh, sorry- you mean "life and death" as in you had a murderbot (https://www.youtube.com/watch?v=xuFbxeDjRZg_ was my team's contribution) that could kill people, and your model prevented it?

Are you sure you can conclude your system is safe?

Nope no murder bots...