I think it is undeniable that both a certain level of working memory and of basic algebra is necessary to understand mathematics at the next level. Someone who could actually understand an "entire college-level course in linear algebra or calculus or graph theory or even combinatorics perfectly well", yet be unable to remember a basic multiplication table, or figure out how to calculate 11*15, would have a very interesting pathology.