> [UPDATE] Let me put this more succinctly: how is it possible that there can be a continuous computable function whose derivative is uncomputable? What exactly is it about this function that makes numerical differentiation fail?

Like this

f(0) = 1

f(x!=0) = 0

It's more of a mathematical trick than anything else, it's not a "natural function" let's put it this way