> Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed.

I get that you'd like to have a clear theoretical basis for what works and we're far from there. But in the meantime we're stumbling in the dark, discovering tricks and forming intuitions, not knowing even where the road is going to lead us.

This is an evolutionary process of ideas, similar to biological evolution that managed to make us. If you know where you're going you can optimise your actions but when you don't even know what might be useful later on, then all attempts are good. They increase diversity and discover blind spots. Some of them will be the stepping stones for the future, but we can't say in advance which and how.

Link to a long discussion about the evolution of ML ideas and the book "Why greatness cannot be planned" by Kenneth Stanley - https://youtu.be/lhYGXYeMq_E?t=416