Agree with your first statement and disagree with your second; I don’t think the former implies the latter.

I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.

These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables.

As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.