The problem is that complex systems need complex design. As your design get more sophisticated, you need more sophisticated ways to communicate the design
I've never found any need for a third, intermediate level of sophistication that sits between code and natural language + informal diagrams.No. Complex systems need more thinking and refactoring until they are less complex.
It's incredibly naive (or at best disingenuous) to assume that complex systems can be reduced by thinking and refactoring. Perhaps some can, but when you consider many of the software artefacts we use daily (seeing as that's the context of this subthread), such as operating systems, virtualization platforms, or even good ol' clunky enterprise applications, you can't just wish the complexity away with a bit of hard work.
I'd be pretty surprised if any of the code that I am using to read this was designed with detailed UML diagrams.
It wouldn't have been possible at all if the pieces that make it up weren't loosely coupled.
>It's incredibly naive (or at best disingenuous) to assume that complex systems can be reduced by thinking and refactoring.
It's naive to assume that you can't. Loose coupling pretty much only comes as a result of extensive refactoring.
Google for "linux kernel diagram"[1]. It's quite well designed, displays loose coupling, high cohesion, and all the other things we like about good complex systems. If I want to dig in and start working on a module, I'd still have to understand the relationship between all the bits I work directly with (and probably some that are further away).
Don't confuse loose coupling with low complexity.
We already have a perfectly good way of representing the details of a system - it's called source code.
Interestingly enough, that's a bit of a false dichotomy.
Imagine it's your first day on the team, and there's thirty classes (or thirty functions) in this software package, and you want to understand their inheritance/composition, what's a good way to take a step back and just see how they relate to each other? You can browse some interface files or some source files, accumulating a model in your head. But the visual throughput into your brain is just so much better when you can see it in a diagram.
Regardless of whether a human draws the UML representation of the design in advance of the implementation or a machine deduces this from the existing source code, you can still benefit from a class diagram that will cut out the noise of the implementation itself. As long as we're drawing a diagram, why not have a common way to represent the concepts we use in software?
Edit: DRY
> perfectly good way of representing the details of a system - it's called source code. ...
> simply use a tool to generate a class diagram from the code
Once you resolve to generate that class diagram you have a choice in how to represent it. UML is just one way, proposed as a standard way, to represent that visually.
The problem of UML is that it tries to be the language of all these levels of design (and even different stages of design!), to the point that the highest-level design document is trying to be something so completely specified that there's no abstraction.
The problem is that complex systems need complex design. As your design get more sophisticated, you need more sophisticated ways to communicate the design. Concepts like dependency, multiplicity, inheritance, inclusion, asynchronous message, exception, timed event are all things you can represent with boxes and lines, but unless you adorn the diagram with a huge amount of text you're not going to capture the distinctions between those different types of relationship.
It's not unlike any other language: I can use the word thought to describe something in my head that isn't physically manifested, but words like lie, memory, hypothesis, idea, belief, image, nostalgia, forecast, imagination are all used to distinguish between types of thoughts. If I used only simple words like "thought" then I wouldn't be able to express anything more sophisticated while still being concise.
In short, if you want to graphically describe something complex, you can either use a sophisticated visual language to do it concisely and unambiguously, or you can use a simple visual language and write tonnes of documentation to support it, with all the problems of attention, ambiguity, and rot that go along with it.