You realized where the mistake in your flow of thoughts is. This is, as I wrote, a good sign.
If you want to become smarter in math: Plan to understand work that some Fields medalists of your choice produced. If available, get a modern textbook treatment of their work (these, in most cases, are better for understandable than their original papers). Of course, you won't understand much at the beginning. So you know in what areas on mathematics you have deep knowledge deficits. So get some good textbooks on these topics to fill these knowledge deficits. Iterate.
-
If you want to become smarter in physics (the following advice is what a good friend of me gave who I really trust regarding this): Start with the 10 books of "Course of Theoretical Physics" by Lev Landau and Evgeny Lifshitz.
> https://en.wikipedia.org/wiki/Course_of_Theoretical_Physics
Having read these books should (according to him) give you at least some very basic foundation of physics on which you can then, as a next step, build by reading much more advanced textbooks.
Honestly: if you come to such a (dubious) conclusion, the conclusion is actually likely true (with respect to you), because if you were smarter, you would sooner or later realize the mistake in this flow of thoughts.