Instead, you should look at how those requirements will be produced. If your architecture "thinks" the same way as your users, it will be able to be modified easily to include more of their thinking. So instead of trying to predict future requirements, you should attempt to capture and accurately model the way your users think about the problem. Sometimes that creates designs that mathematically/algorithmically seem pretty ugly. But in the long run I'll often discover that the ugliness exists for a reason--people mostly only add complexity to their mental models because that complexity allows them to solve a problem. So even when the domain expert's thinking seems overcomplicated, it's usually actually the least complicated model that accurately represents the problem.
Rather, listen to your users problems, listen to what they think a solution is, then let your design team determine if there is a better and more simple solution. Sometimes there may not be, but often there is.
When you get requirements from your users, they're going to mix in how they think about the problem with how they think that problem should be represented in the program. The problem with that is that they don't know what computers can do, so they can't possibly be expected to know how the problem should be represented in the program. And in fact, the UI, which is the only thing they ever see, will probably be more task-oriented than representation-oriented, so they probably will never see how your program actually models the problem and therefore won't be able to suggest good changes to it.
Ultimately, though, there's an underlying mental model that the user is using for the task. And for most fields, there's really only one correct mental model, so I actually disagree that expert users have different mental models, at least not with significant differences.
Quite often in super early projects, the domain is not understood, so the architecture reflects the learning (or misguided assumptions).
It's not about that, it's about keeping what you do nimble enough to always be ready for change when it comes. That means not having duplication in the system, having things well factored and encapsulated so there's a distinction between the abstract and the implementation so at any time any component can be made a change point and become pluggable. What makes a good architecture good is that it can be changed without scrapping the whole thing or breaking the whole thing.
> Where good people win is that they adapt quickly when there are signs that the design doesn't work.
It's not that they do adapt quickly, it's that they "can".
> A lot of people just "stick to the plan" although everybody knows that it won't work.
Because they're stuck with a bad design that can't be changed due to issues like bad factoring, lots of code duplication and copy pasting, etc.
From my experience they just don't try. There are (almost) always ways to improve.
I have a few enterprise projects now over 10 years old that began as simple projects. It took having enough understanding of the domain and problem set and then anticipating what was expected to grow and change, and what did not.
No design is complete in the start. It needs to find a balance between being a foundation for exploring and developing through the problem domain, but remaining flexible. I think a lot about managing and mitigating risks through clever and simply designed architecture that can do a lot. The less there is to the design the more it can remain flexible to deal with new facts.
The fine balance of over architecting/abstracting vs under abstracting/architecting is for me a matter of finding the balance between the size, impact and possible number of unknowns about what I'm trying to design when I begin vs the future. Where possible, understanding the domain before daring to design or interpret is often the most important factor.
For the few large enterprise systems I've designed in unrelated fields in the past 10 years, I haven't ended up throwing away or majorly refactoring the schema or design. What did change is the stack and tools that became available and making sure I kept a laser focus on an n-tier architecture so different layers of the presentation, business logic and app logic could be swapped in and out over it's lifetime. I guess I can say comfortably that risk of designs falling apart was minimized, and luck has increased because I didn't begin without an understanding of the core principles, entities, and interactions ultimately the details and workflow the system has to manage.
I can't stress the importance of keeping things as simple as possible, because complexities are guaranteed to arise in the future.
To me, features, and schema changes over time can become like perpetually flying bullets.
I absolutely have refactored or adjusted as I go? Sure. Does all of it get thrown away? Rarely. Have I over-architected a solution to cover my butt and the new facts never came, totally. Has a new tech, stack or library come out that made me need to replace what I had built or modified to do what I need, always.
I do lots of paper models to try and see the core patterns, and what could be interpreted in a number of different ways should something grow in that direction. This type of strategy is far more difficult when working in a problem space where there is a lot of research and development.
How about you? Would love to hear how you approach things that have different balances of known/unknown facts.
Where good people win is that they adapt quickly when there are signs that the design doesn't work. A lot of people just "stick to the plan" although everybody knows that it won't work.