On another thread, there was a recommendation for Roger Penrose's 'The Road to Reality'. Haven't read it, but it purports to build from basic math through to advanced concepts and physics, step by step.