You could still verify some things, e.g. if you expect the model to have a certain kind of symmetry, or always produce outputs within a certain range, or something like that. Of course it would be best to encode those expectations in the model structure or training algorithm, but it might be useful to know whether you got it right.

For the common case of "The model misclassified a data point, no idea why.", formal verification doesn't help, but that doesn't make it completely useless. Probably not worth the effort, though.