You will also need a pretty good understanding of calculus and linear algebra to understand QM, which many CS people don't have. You could probably learn both at the same time, but it would be challenging. The unfortunate thing about physics is you need to have pretty strong math skills to understand things, and most people's mathematical preparation is not very good.