I might be downplaying CS programs, I just experienced one of them. In that program, I had no chance to learn compilers, operating systems, electronic circuits, assembly language, or any math. In hindsight, I'm not sure if this was the right choice. I did learn about those topics, but just in my spare time. I was able to pass my CS program while barely going to lectures (which may sound nice, but in practice I did not make any friends; it was a terrible time). I could just as well learn this on the side while studying something that is intrinsically more interesting, like physics or electrical engineering.
At the other hand, I was able to double major, and it's not hard for me to find a job now.
I found a very sweet spot in Embedded Systems programming, which by nature is more or less an intersection of the two disciplines.
As a self-taught programmer, I think you're downplaying how much you can learn in a CS program. I don't pretend to know as much as a CS major. I haven't spent time learning compilers or DB internals, and probably haven't studied CS algorithms enough to pass an interview at Google or Facebook.
Regarding the CS curriculum, an undergrad EE curriculum is similarly shallow: it's a sampling of many different subtopics that often are only loosely related (E&M, DSP, circuits, control theory, devices, etc.). Undergrads won't be experts in any one of those fields.
Since I graduated (grad school) in EE, I've told myself each year that I'll jump from my EE job to a programming job, since I enjoy programming more. But it's getting harder and harder to justify. Will I be leaving a senior EE position for an entry-level software position? Does that make sense at 30? Why am I the oldest person in the office when I interview at a startup? Why is the interviewer fixating on compilers and database internals?
If I were to do it over, I'd get a degree in CS because that's the most direct route to the job, and education continues on the job. Everything else was a diversion. And you're seeing the grass as greener on the other side of the fence regarding engineering/physics curriculum. Learning how to solve physics problems doesn't make you smarter or a better programmer.