But as long as you know the math it is easy to self learn the other parts of computer science. I just studied physics and math in college and self learned the rest and learned algorithms, compilers, distributed systems etc on my own and have worked on those at Google and never had a time where someone knew something they learned in CS that I didn't know. It was the opposite, since when I self learned I went through every CS topic you can take I often knew a lot of CS things that the CS grads didn't know since they were limited in how many credits they could take.

I know many who took a similar route as me, but I've never met anyone who self learned math to a reasonable level. And since math is a very small part of a CS degree you could learn all of it in 1-2 semesters, so the optimal way to learn computer science would be to take 1-2 semesters of college math and spend the rest self learning. I bet you'd get a better understanding that way in much less time than it takes to go through a CS degree.