Programming and math _is_ essentially the same (Curry Howard isomorphism) and the problems are indeed equally complex.
The difference is that programming is driven on economical terms, hence agility, flexibility, etc. has been developed.
Mathematics is driven in the university sphere, where mostly intrinsic motivation drives. Not many mathematics professors have the urge to sit down and learn scrum, read manifests of coding practice etc.
The math culture will eventually see itself be forced to go these ways to keep up.
Also, while programming, you can design your functions and their interfaces before writing them down. In maths, that's impossible: you're constrained by what can/cannot be done.
Scrum or coding practice doesn't apply here. Maths don't have to "keep up", they're already way ahead of their applications. It also doesn't need to be fast or flexible, just rigorous. And while motivation and intuition helps (specially when learning) some concepts cannot be motivated or given an intuition, and even in that case you still have to learn the small details and formalizations.
Have you forgotten the pains that you went through trying to understand the difference between values, pointers and references, lexical and dynamic scoping, static and dynamic typing, and the like? Can you see them with your eyes? Have you ever tried fully explaining any of those to a non-CS major in half an hour? :)
Abstractness is pretty subjective. For non-programmers, even the idea of CPU and memory can be abstract. And it doesn't help to open up a computer and point to the hardware; that is like claiming a mathematical paper is not abstract by pointing to the very concrete paper and ink that embodies it.
Some concepts in mathematics are way above the abstraction level that you mention. For example, the projective plane, or nowhere-differentiable functions, or geometry in higher dimensions; those are concepts that not only do not have any physical equivalent, but are also very difficult to grasp and imagine in your head.
Would it actually work? I honestly don't know, but I'd seriously love to try it.
Math _indeed_ have the idea about interfaces and implementations. They are called something different (exitensisal types from a PL point of view).
I will recommend you to look into type theory, in particular the book "types and programming languages" provides a very good introduction.
In CS we have two areas that make it completely clear that math and CS are coupled, that is complexity theory and programming languages. After having taken a course in each, and pondering a bit, it should indeed be possible to see the that they are the same.