I made it thru my interviews without remembering obscure things, only knowing basic O-notation, and generally not being a math genius.
The majority of interview questions I've encountered (at Google and elsewhere) are of the form: here's a data structure or API, can you use it to solve some problem? Nobody's ever asked me to prove A* from scratch. Even in cases where the question does revolve around a specific algorithm or CS concept they've always taken the time to explain it. I used to ask a question based around LZ77 encoding and I always went over the algorithm whether or not the candidate was familiar with it.
It is very different to attend a calculus class and learn to apply some formulas and that's all. You vaguely remember the formulas some years later, and the algebraic abilities are all you have, but they are still very powerful tools.
However, if you do a mathematics course like advanced linear algebra then you realize that for this particular teacher knowing the formulas doesn't matter, at all. You have to write proofs for everything. And these proofs require all of your imagination, all of your experience and then some.
It changes the way you approach every single problem from that moment.
Now, most people have never needed that ability. Most people will never exert themselves to the stress, the lost nights of sleep and the sheer single mindedness that drives everything out of your mind for days, until you get that proof. That damned proof you will never forget for months, if ever.
The end result is that it makes your mind work in overdrive for solving the same kind of problems.
So the issue is that some people have memorized stuff (and that never works well for too long), while other people has taken that stuff as the first step, and then through a long process of proving every single step of one or more theorems, have tattooed that stuff in some part of their minds, and are simply trying to find someone having a similar tattoo, to find someone with whom they can relate about their shared experience.
Anyway, the people I know that can do that, don't really like to program computers.