Not as a rule exactly: these days, anyone reasonable savvy person can learn CS well enough to be employed in tech and any math-savvy person can certainly end up in CS research. But it is, I think, _super_ hard to start learning programming in college and actually end up good at _low_-level computing like Kernel/Systems level stuff, which really requires like total immersion in the space, similar to learning a language by living in the country. In my program this meant that the really technical classes were filled with the students who came in already very computer-literate and into programming, and they tended to be 90+% male.
(There are exceptions. I'd characterize Charles Severance's MOOC from UMich as a true beginner class. But, by contrast, I think I'd find MIT's intro class in the CS department (6.001) impossibly difficult if I was having to basically learn to actually learn simple programming on the side.)
[1] https://www2.seas.gwu.edu/~kaufman1/FortranColoringBook/Colo...
[1]https://www.aei.org/carpe-diem/chart-of-the-day-the-declinin...
If CS was mandatory from grades 1-12 then things might be a lot less skewed in those first year introductory CS courses. Of course you're still going to run into the issue of kids who take up programming as a hobby at a young age and arrive at university with a large lead in skills. On the other hand I've seen math students who have done the same thing.
I actually did well in math throughout high school but it wasn't a particularly strong skill of mine in university. And I could have no more majored in math (or physics) than I could have flapped my arms and flown.
My secondary point still stands. Just as there are kids out there who grew up programming in their bedrooms who come into first year CS and crush it, there are kids out there studying extra math and writing math contests etc who will come into first year with a big advantage.