it's still an extremely valuable tool. just as we see in mathematics, closed forms (and short and elegant proofs) are much coveted luxury items.

for many basic/fundamental mathematical objects we don't (yet) have simple mechanistic ways to compute them.

so if a probabilistic model spits out something very useful, we can slap a nice label on it and call it a day. that's how engineering works anyway. and then hopefully someday someone will be able to derive that result from "first principles" .. maybe it'll be even more funky/crazy/interesting ... just like mathematics arguably became more exciting by the fact that someone noticed that many things are not provable/constructable without an explicit Axiom of Choice.

https://en.wikipedia.org/wiki/Nonelementary_integral#Example...

>closed forms (and short and elegant proofs) are much coveted luxury items.

Yes, but we're taking about roughly the opposite of a proof

but in usual natural sciences we don't have proofs, only data and models, and then we do model selection (and through careful experiments we end up with confidence intervals)

and it seems with these molecular biology problems we constantly have the problem of specificity (model prediction quality) vs sensitivity (model applicability), right? but due to information theory constraints there's also a dimension along model size/complexity.

so if a ML model can push the ROC curve toward the magic left-up corner then likely it's getting more and more complex.

and at one point we simply are left with models that are completely parametrized by data and there's virtually zero (direct) influence of the first principles. (I mean that at one point as we get more data even to do model selection we can't use "first principles" because what we know through that is already incorporated into previous versions of the models. Ie. the information we gained from those principles we already used to make decisions in earlier iterations.)

Of course then in theory we can do model distillation, and if there's some hidden small/elegant theory we can probably find it. (Which would be like a proof through contradiction, because it would mean that we found model with the same predictive power but with smaller complexity than expected.)

// NB: it's 01:30 here, but independent of ignorance-o-clock ... it's quite possible I'm totally wrong about this, happy to read any criticism/replies

Isn’t that basically true of most of the fundamental laws of physics? There’s a lot we don’t understand about gravity, space, time, energy, etc., and yet we compose our observations of how they behave into very useful tools.

> what have you really discovered?

You’ve discovered magic.

When you read about a wizard using magic to lay waste to invading armies, how much value would you guess the armies place in whether or not the wizard truly understands the magic being used against them?

Probably none. Because the fact that the wizard doesn’t fully understand why magic works does not prevent the wizard from using it to hand invaders their asses. Science is very much the same - our own wizards used medicine that they did not understand to destroy invading hordes of bacteria.

Exactly! The magic to lay waste to invading armies is packaged into a large flask and magical metal birds are flown to above the army. There the flask is released from the birds bellies and gently glides down. When the flask is at optimum height it releases the power of the sun and all that are beneath it get vaporized. A newer version of this magic is attached to a gigantic fireworks rocket that can fly over whole mountain ranges and seas.

Do you know what the stories say happens to wizards who don't understand magic?

https://youtu.be/B4M-54cEduo?si=RoRZIyWRULUnNKLM

The how is actually just further hypotheses. It's turtles all the way down:

There is a car. We think it drives by burning petrol somehow.

How do we test this? We take petrol away and it stops driving.

Ok, so we know it has something to do with petrol. How does it burning the petrol make it drive?

We think it is caused by the burned petrol pushing the cylinders, which are attached to the wheels through some gearing. How do we test it? Take away the gearing and see if it drives.

Anyway, this never ends. You can keep asking questions, and as long as the hypothesis is something you can test, you are doing science.

>There is a car. We think it drives by burning petrol somehow. How do we test this? We take petrol away and it stops driving.

You discovered a principle.

Better example:

There is a car. We don’t know how it drives. We turn the blinkers on and off. It still drives. Driving is useful. I drive it to the store

In the vein of "can a biologist fix a radio" and "can a neuroscientist understand a microprocessor", see https://review.ucsc.edu/spring04/bio-debate.html which is an absolutely wonderful explanation of how geneticists and biochemists would go about reverse-engineering cars.

The best part is where the geneticist ties the arms of all the suit-wearing employees and it has no functional effect on the car.