Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.
The point here is that we are descending in a continuous and monotonic loss function. You don't need an accurate derivative or even an estimate of the derivative to do so, although it definitely helps a great deal.