Formally, there is a generalisation of differentiation which can handle functions like ReLU (i.e. locally Lipschitz non-differentiable functions) by allowing a derivative to be set-valued. It's called the Clarke gradient. The Clarke gradient of ReLU at 0 is the closed interval [0,1]. Note that the Clarke gradient doesn't satisfy the chain rule (except in a weakened form) which might seriously mess up some assumptions about autodiff. Is this generalised derivative useful in autodiff?
I imagine that this is a largely theoretical tool that's useful in analysing algorithms but useless for actually computing things.
[edit]
Question: Are there numerical applications in which the subgradient is actually computed, or is it a purely analytical tool?
See for example : https://www.stat.cmu.edu/~ryantibs/convexopt-F15/lectures/07...
and related neat usages in set based optimization methods in MathOptInterface (part of JuMP.jl): https://matbesancon.xyz/post/2020-12-24-chains_sets2/
The gist of it is that we endeavour to provide ‘useful’ values for the gradient at non-differentiable points, even if the traditional derivative is not defined or infinite.
You could imagine it as us smoothing out edges or discontinuities, ideally in a way that that makes things like gradient descent well behaved.