I followed a similar path and can only agree on the attitude. Most of the relevant CS knowledge is just that: knowledge. Studying physics is usually less about knowledge and more about general problem solving skills.

I'd go beyond "general problem-solving skills". Physics gives you high quality math training (albeit somewhat truncated with respect to number theory), coupled with demanding experimental methods, and a deep, principled understanding of the foundations of the universe. Nothing is beyond your reach. Any other quantitative or logical discipline is just laying there, waiting for you.