It's called "systems analysis". Programmers are generally pretty terrible at it because it requires holistic, big-picture thinking. But it used to take up the bulk of the design activity for a new enterprise system.

And the result was usually a complex system that no one needed and could maintain. Successful refinement is needed when you try to accomplish something from the ground up, refining it as you're adding features. Not locking yourself in the ivory tower drawing UML diagrams and drafting requirements. Doing and thinking go in pair, not separately.

Yeah, the field of software engineering has come a long way since then. But just because previous implementations of the analysis phase were flawed doesn't mean that the phase itself was flawed.

Research conducted by M. Bryce and Associates suggested that use of a structured systems analysis phase before programming began resulted in time and cost savings vs. just "hacking it together" like programmers want to do. Locking yourself in the ivory tower is an unfair way to characterize systems analysis. Systems analysts talk to people in the business to understand what the business requirements actually are, and then design an information system (NOTE: not a computer system or software system; if it cannot in principle be run on pen and paper it is not an information system) that meets those needs. Programmers only come in when the automatable parts of the system need to be implemented, and work from a detailed and precise spec.

> Programmers only come in when the automatable parts of the system need to be implemented, and work from a detailed and precise spec.

A lot of systems are complex enough that you can’t get to that stage (and if you think you do, think again). Mostly because of communication issues and time concerns. Which is where the agile manifesto comes in and recommends the talk-do-show in short cycle. It needs not to be hacked together and the showing helps with communication, the talking guides the doing, and the doing is what pays for everything.

"Show me your code and conceal your data structures, and I shall continue to be mystified. Show me your data structures, and I won't usually need your code; it'll be obvious."

Lots of people try to make their code better by revising their procedures and don't even think about changing their data model, and generally fail. You might not be able to change your data model without changing your code, but they're different activities.

Or not using the data model properly, zero foreign keys in databases, no triggers checking column contents etc.

"We'll do it on the app level".

sigh

No, that part is just smart. Databases have terrible support for nontrivial datastructures, doing everything at the app level is the only reasonable response.