> If we know these numbers (and nothing else), we can linearly approximate the values of a function near a particular point, with at most quadratic error.

Ignoring the fact that you need the function's values actually at the particular point in question, you do in fact need to know something else: you need to know that the function in question has sufficient regularity (enough smoothness) to allow an application of Taylor's theorem.