A mathematical proof or analytical solution. I understand that analytical solutions only apply for very narrow ranges of problems but... consider writing an analytical solution to a differential equation versus applying a numerical solver. A numerical solution rarely leads to as deep an understanding as an analytical solution or a number of approximations in various limits. I feel like we're in the limit of something analogous to writing numerical solvers and claiming understanding from observing the output.
If it's the former then this is part of building it. An implementation without proper testing is incomplete. If it's the later I actually agree. Only the most sensitive of applications require that level of sophistication though.