There is already active research on this. Clark Barrett has been applying SMT to deep learning models to prove smoothness guarantees.