On the one hand, there are problems which by accident or design are nondifferentiable. Cryptography, for example.

In the other hand, these problems are routinely analyzed and solved by differentiable algorithms running on neural net substrates (e.g. you).