Finding an optimal solution to explore-vs-exploit is the "multi-armed bandit" problem.

https://en.wikipedia.org/wiki/Multi-armed_bandit

> Originally considered by Allied scientists in World War II, it proved so intractable that, according to Peter Whittle, the problem was proposed to be dropped over Germany so that German scientists could also waste their time on it

...though it's actually gotten a lot of great mathematical solutions over the decades :)