Come to find out, he was a math prof, and very much not an algorithms guy. There are two versions of big-oh notation: one for algorithmic complexity, and one for accuracy of approximations. They are exact opposites of each other: higher numbers are "better" for approximation accuracy, at least in the sense they were used in the course. He was unaware of the former, but both of us were unaware of the latter.
Not such a great teacher, no. Still, the last mistake was an entirely reasonable one. It's a great example of how even math students need to be on their toes.
1) Didn't have a solid understanding of algebra. and 2) Didn't even learn all of the algebraic concepts.
I was fine (because I could recognize the concepts even though he made a TON of mistakes) and skipped (precalc/) trig (which I learned during..) and went right on to AP calc . But the point is that the fact that the teacher made so many mistakes led to the students not being able to comprehend slightly higher level math. (and him being fired)