Lots of people make bad designs today, sure, but it's not for a lack of formal methodologies, because average design quality was way worse back in the design patterns 'n' UML days.
I got out after butting heads with him constantly, and I don't think said company ever shipped anything meaningful in his entire tenure here.
When I worked there I was a product guy, and it took me an embarrassingly long time to understand why I didn't fit in... but yeah, I didn't last long.
For anyone who didn't live through that time, the Open-Closed Principle states that software should be open for extension, but closed for modification.
However, you could also rephrase that principle to be: 'you should always prematurely abstract your designs'.
I think if abstraction was viewed as a negative to be avoided unless necessary, software architecture would have been far better off.
To be fair, premature abstraction is a lot of fun for those that do it. It's just those that follow who aren't so keen.
E.g. I don't create an abstraction in case I someday need to switch the database from SQL to NoSQL, but when I need the abstraction right now for an alternative implementation (e.g. mocks for testing).
Every class has an interface abstraction, or worse inheritance hierarchy, that adds friction to change, even though it's the only implementation.
When something similar comes along it's pressed into that shape, because the abstraction is already there.
It's like this meme video where it's hard to watch all the round and triangular pieces being fit in the square hole.
And the problem is that while software is more flexible than traditional architecture, hence soft, the complexity limit is also soft or virtually non-existent. So in software it's worth more to be simple and flexible than to have a plan that's long and detailed.
DDD Europe by all accounts was a resounding success. We’ve replaced older architecture activities like data modeling first with event modeling and Domain-Driven Design.
We’ve learned to embrace monoliths when appropriate and reduce complexity with bounded contexts.
We have phenomenal testing capabilities that didn’t exist 20 years ago.
We have a myriad of data storage tools.
We understand front-end engineering from an information architecture perspective.
We can design detailed architectures in tools like LucidChart very quickly with detailed solution details.
I’ve been at this for 40 years and I’ve never felt better about being a software architect.
I do however agree that design is sorely missing from the current software project management zeitgeist, which means it’s done in a more ad hoc way. People are taking frameworks like Scrum far too literally, and I agree that in some cases there is little vision or overall architecture because the framework doesn’t include these things. There should be design and review activities both before and after coding, but these are largely neglected from most project management frameworks. Scrum for example doesn’t even include backlog management, which in my work is a critical activity.
While that doesn’t mean design, architecture and scope management is not done, it’s certainly less visible than it might be.
All of that said, I’m a big believe that the ultimate expression of any design is code. The design artefacts that come before code is written are scaffolding, age quickly, and are soon useless. So there’s not much need to keep them around, and they are mostly of interest as preliminary directions.
In this regard I’m a big fan of Jack Reeves essays from the same period at https://www.developerdotstar.com/mag/articles/reeves_design_...
Like many abandoned corporate practices, I think it was abandoned for good reason. It may have made sense under different circumstances, like when you had a large army of cheap offshore devs who could not be trusted to architect a maintainable application.
If I had some ivory tower “architect” trying to interfere with my work I’d be so pissed. Anybody I’ve seen with that title, that wasn’t doing 1P cloud consulting where the title means something different, usually had no clue what they were doing and had been given the title as a soft retirement.
It is pretty funny when somebody runs into me and realizes for the first time that I have the job because I build stuff and write code, not because I'm good at LucidChart. I'm planning things out beyond immediate needs, but not because I'm looking for job security--it's because I've built the thing we're doing before and would like to not make the same mistakes I've made in the past. I'm over here demanding adequate standards of code and low- to mid-level design, and the "wait he's serious?" of it is sometimes honestly pretty fun to run up against.
I am good at LucidChart though.
Principal engineers and such who are still involved in operations, implementation, and more tactical approaches are who I also think are “supposed” to be doing architecture, but even then, more as guides and first among equals than as people who hand down decisions from on-high.
The fundamental issue I have with a separate architect position is that it disempowers teams and makes them beholden to decisions that they may not agree with (and which very well may not understand the problem to the extent they do). It sounds like you’re doing the better thing of running up and down the layers of abstraction so your contributions empower people rather than disempower them
This, I'd agree with. You have to be at the coalface to know what the hell is going on. At the same time, you have to be cognizant of business needs and why things are the way they are, which is to me a fair approximation of "the job of a principal engineer."
(My other hat here is "head of API governance" and that's largely a business-flavored analysis of APIs being brought onto our company-spanning platform. I couldn't escape having both in my head if I tried.)
> It sounds like you’re doing the better thing of running up and down the layers of abstraction so your contributions empower people rather than disempower them
Ideally, yes. In reality, I work for The Phone Company, and The Phone Company hires a lot, and I mean a lot, of vendor devs. I am doing their thinking for them a lot of the time; the swerve is that I can and do write code (have released moderately popular open-source libraries on their framework of choice, for example) and so the usual development practices of "sure let's make a dozen packages for marginal functionality" don't fly.
I am disempowering them, because ultimately, we will eventually be cycling out our vendors and I will be the one who has to own their output. So that output has to be something I can live with. But this place is Processes Georg and should absolutely not be counted.
(I like the job. I will enjoy when I eventually go back to a shop where the developers have a reason to feel ownership over the work.)
i prefer to reason about a system and component relationships using, say, a single word as representation, instead of glaring at one or many inheritance directionals, interface details, and other "field" information, which is usually conveyed in a UML node.
i do not think the lack of formalism is a degradation -- we work with abstractions after-all, so it makes sense to further leverage that fact and express things simply, at a high-level, and straight-forwardly -- you can pack a lot into a single word.
of course as the nature of any tool, there is a time and a place for its application. but i don't think it is fair to call it a degradation.
The much worse outcome is "pick a thing, fiercely defend it whether it helps the ship to float or to sink".
I was eager to use UML when it came out but I agree with the article, the only thing I kept was sequence diagrams. Most of the rest of it was just a complicated way to represent stuff that is best represented in code or plain documentation.
I’m also not afraid to use stuff that’s out of fashion. I use simplified ER diagrams, and even flow charts on occasion.
But the tool I use most often in design is SQL DDL…
"The following essays by Jack W. Reeves offer three perspectives on a single theme, namely that programming is fundamentally a design activity and that the only final and true representation of "the design" is the source code itself. This simple assertion gives rise to a rich discussion—one which Reeves explores fully in these three essays."
https://www.developerdotstar.com/mag/articles/reeves_design_...
Teams aren't shunning design. They're just doing it incrementally, because as it turns out, trying to design a piece of software before the first line of code is written doesn't actually work.
Even in peak UML (2000 or so) there were no really practical UML diagramming tools. (Rational Rose was impractically expensive, and buggy enough that it was a race to see if it wouldn't crash before you actually finished your diagram).
Corel Draw had passable UML diagraming support. But I don't imagine it still does.
Draw the boxes and lines and make sure that people understand what they mean. Describe the system from the perspective of the reader. Just like a real architect will have different drawings/plans/elevations for their customer, the planning authorities, and the builder.
The architecture diagrams are meant to communicate the design, not comply with some over-worked standard.
UML was a clusterfuck that evolved from the trifecta of late 90s OOP (inheritance not composition), design patterns that mostly provided fill-ins for what was missing from Java as a language, the ridiculous concepts of generating code from diagrams that could be regenerated from code, which never, ever, worked, in the same way that ORMs have an impedance mismatch between OO and relational logic.
It was yet another silver bullet, the late 70s/early 80s had "structured programming", the late 80s/early 90s had CASE, the 90s had all the stupid diagramming tools where people argued about the shape of the bubbles and what arrows to put on the lines.
There was also the Capability Maturity Model, where everyone was trying to get to "Level 5" which was only useful if you were doing exactly the same software over and over again, along with the "6 Sigma" and "Black Belt" nonsense.
The 2000s had the "iterative Rational Unified Process" (an excuse to sell expensive tools from IBM), along with CORBA et al.
The last decade has suffered from the Agile-with-a-capital-A and especially "Scaled Agile" which is just an excuse for project managers to again treat programmers as fungible, while losing all of the affordances of actual project management, like GANTT, critical path, S curves, earned value etc.
Sprints are nonsense, so is "t shirt" sizing and velocity and burn down charts, and retrospectives and scrum "masters".
There are a couple of useful things:
* Domain Driven Design, using Names That Make Sense To The Customer
* Entity Relationship Diagrams, where the Entities are the same Names as the DDD
* State Diagrams for those Entities, describing the events that will cause them to change state
* Event definitions that match the State Diagram transitions, where externally generated events end up being your external API.
Bah, I've been in this industry for almost 40 years and its the same shit over and over, just with different names, and different consultants selling expensive courses.
The risk and complexity dictate how much architecture and engineering are involved and with all things, businesses will try to get by with a bare minimum. The automotive, chemical, and other industrial sectors invest a lot in software architecture and enterprise architecture because the risk is so high. But even a medium size e-commerce platform has very low risk or complexity.
That sentence is bringing back ptsd-like flashbacks.
I remember distinctly how much everyone thought that approach you describe was broken when it was the trendy thing.
I'm not sure how universal it ever was, but it was at one point a trendy thing that everyone thought they should be doing... and that so many people subject to found disastrous. [The phrase "architecture astronaut" was a common epithet, and not a friendly one]
That above paragraph [without the brackets], ironically, could certainly be said of "agile" more recently too. I don't know how universal it ever was, but it was at one point a trendy thing that everyone thought they should be doing... and that so many people subject to found disastrous.
But yeah, I'm pretty convinced going back to appointing some almighty deity architecture astronaut who isn't responsible for or involved in any implementation (let alone operations!), who hands down plans from on high after "memorizing all design patterns" and drafting some diagrams, never sullied by "contact with the enemy"... no thank you, but thank you.
----
Instead of just complaining about that though, what I'll say in addition is -- I think the real problem is that engineers aren't given the time to carefully consider top-level designs. it's a basic business/resource issue -- until engineers have more breathing room to talk to each other and research and consider and come to decisions in an unhurried way, the top-level design stuff will remain chaotic. It's not an issue of appointing an ivory tower "architect", or something solved by it.
Although sure, there should be senior and even "staff" or "principal" people with more authority/responsibility for higher-level designs.
Everyone should be responsible for design at the level they are working. Everyone needs enough time to feel like they are doing it well, instead of running on a sweatshop code production treadmill.
That said, I've been at places where management did a pretty good job of making sure that there was enough time to do this kind of work with middling results. People have to enjoy the work or feel invested in a way that makes them care about "3 years from now", and sadly in a lot of places there's a lot of "whatever, it'll be fine for as long as I work here."
It seemed to me that in let's say 50% of cases, having the extra time didn't really matter. Assumptions about what was being built were proven incorrect not long after a design was laid down and work had begun, or particular engineering types would create increasingly complicated designs with their time rather than doing the hard work of distilling the problem, sometimes including $PET_TECHNOLOGY as part of their solution.
The "let's design something truly great" really has to be in a team's DNA in my experience or you wind up with a good design that isn't followed, one person who does all the work, a gold plated design, etc.
I'm not sure what to make of it as far as general narrative.
If people are creating stuff they don't really care about, of course it won't be very good? And there's a lot of money spent paying people to build things that honestly nobody would reasonably care much about?
One way or another, the pining for a software engineering world where we all collaborate on creating things that are maintainable products with high-quality user-facing experiences, and have the time to do that... well, that's not the one we've got.
If I went through and inspected any Node, Rails, Django, etc app I would find many Gang of Four design patterns, but very few of them would be in the project-specific code. They got implemented well, and now programmers can build new things that would have taken too long to do before.
And that was the intent of the Gang of Four book, not to teach you patterns you should copy mindlessly, but to give examples of how to identify and extract useful patterns from the software you've already written and describe them to others. Since that is a lot more difficult than memorization, very few took up that work.
Design patterns are what let you deal with the mess that results from that.
Besides, Peter Norvig, even as of 1996, is not not the sort of neophyte you’re referring to.
It's possible to understand design patterns and work in a language that doesn't require them as much.
Here's a talk about it from 2009 with regards to python.
I think in some ways AspectJ can make the observer design pattern obsolete in Java. I have arguments against doing it in aspect way, but it was revelation nonetheless.
Back in the day (before open sourse become so prevalent), lots of software was designed from scratch with the minimal use of outside components and frameworks.
Today OSS frameworks (especially web frameworks) and libraries, PaaS, and Cloud provides you with an already baked-in design patterns. So there is less need in SW Architect and proper design.
Also, most of the Gang of Four patterns are just addressing the deficiences of OOP and older PLs, so if you're using a modern PL with closures/etc. there is no need in them.
On the other hand, Boeing's crappy software practices resulting in the 737-MAX crashes are an area where short-cuts killed people.
The same market that gave us AT&T, Comcast, DRMed IoT Juice presses, FTX, Enron, and so on? Not sure it’s wise to conclude that the market produces optimal solutions, or even incrementally better solutions than yesterday. Especially in domains with extremely long feedback cycles, like organizational trends.
I DO think there is some hocus pocus market zeitgeist that does something resembling gradient descent, but it’s acting over very long cycles and there’s a ton of room for cargo culting, grifting, and opportunistic grabs along the way. Heck, marketing is an entire field dedicated to affecting “rational actors” in the marketplace.
I agree. I don't personally fetishize the old slow processes nor ivory tower architectures, but I think there's a healthy spectrum between that and the complete 'software fuckery' (https://web.archive.org/web/20160308032127/https://medium.co...)
I think what many people, esp. from outside the SWE world, don't get: Software engineering is a deeply social kind of work. There are dozens of solutions for the given problems, you have to agree on one that works for all peers. That's the job. Optimizing it for drawing funny diagrams is not an issue if not for communication.
If you study CS in a German university you can easily get an MA without being able to write software at all (I personally happen to know several people). German universities teach what is easy to teach top down and test for: The textbook stuff that came out of the whole Java EE/OOP/SOAP/UML sector. You barely get practical coding lessons and can avoid them completely if you want. The academic sector never realized how crappy German software products are and never bothered looking at what Big Tech is doing. With given data protection and soon AI regulations, as a university you'd have a hard time collecting enough training data because your law department would step in referring to the current legal insecurity (I've heard stories from friends).
Then we have this little crazy island Berlin which up until maybe 10ys ago was mainly driven by the infamous Rocket Internet "startup incubator" which is led by a couple of MBA sociopath billionaires, trying to copycat everything from SV and then sell it back to the SV company whenever they wanted to start conquering Europe. Thing is they never really developed enough SWE excellence to get the copycat successful in Germany or anywhere in Europe (with some exceptions).
Third example? Here you are: Today I learned that the gov't already decided 20ys ago that they want to provide all usual governmental services online. 20ys later they (allegedly) poured 3.5Bn EUR into an unholy setup of consulting businesses, incompetent civil servants and a panel of software architect astronauts who could never really agree on things. All their deliverables are click-dummies, gazillions of PDFs with SOAP/WSDL/OMG/UML thingies and prototype projects rolled out in "experimental" cities. So if you happen to live in Bremen you might be able to register your dog online but not in Berlin. Therefore in Berlin you might be able to get a license plate for your car online. Pretty much all governmental projects (Covid vaccination registration, special governmental aid for students because of high inflation, etc.) broke down because all their systems are incapable of handling more than maybe 10k visitors (in my theory it always breaks down whenever the biggest single Oracle DB host they could buy is going down).
Germany has some decent software engineers, especially if they're self-trained and not brainwashed by one of the universities or big corps. But the environment manages to regularly piss them off and make em emigrate to somewhere else.
Ouff, much text. Hope at least someone enjoys reading it.
---
[0] This is because if you start a company in Germany you're faced with horrible bureaucracy wrt taxes, laws, politics, governmental authorities, etc. For example, you're forced to pay for a membership in a funny non-IT institution called "Industrie- und Handelskammer (IHK)" which essentially consists of a crowd of old men who are officially supposed to lobby for you and create a networking environment but if you ask them something like "hey, can you tell me how many companies are having problem XYZ right now?" they will tell you that they don't have any numbers and have no means to collect them. In 2023 they still send out a meaningless paper printed magazine. So not helpful at all but take a significant share of your gross turnover mainly to pay for their pensions. Additionally, with all the regulations the governments set up over years they're now facing a significant cut in civil servants because Germany is getting older and older. As a result they're not having enough people to enforce or check regulations in time and never managed to develop any IT-based systems. Big problem with the influx of refugees in recent years and affects many other concerns as well. Finally, there's this cultural difference to, e.g., the US that average Germans are not business-savy at all. If you tell the average German mom that you want to start a business, she will tell you that you're a dreamer and should get a proper job. Germans generally tend to think that companies are something god-given.
[1] They are still the go-to businesses if the gov't quickly needs something, like the Covid tracing app which German tax payers AFAIR ended up paying 120M EUR for (lol).
As for the old people running the IHK, from what I read, the membership is one-company-one-vote, so what stops people running for election?
As for the influx of refugees, that is a distinct advantage to Germany of an increased availability of workers, including many educated Syrians and others.
So it sounds like there's a problem with entrepreneurship in Germany outside the engineering / petro-chemical businesses. There's also a problem with your political choices due to the usual issues prevalent in every Western country. An aging population, the effects of the "financial industry" (a misnomer if ever there was one), the effects of climate change, etc.
This wastes a huge amount of time and usually the documents become obsolete as soon as you try to actually run some code.
This is the very issue the Agile manifesto identifies and proposes a solution to when it says "working software over comprehensive documentation". 'Comprehensive' is a key word as they don't mean NO documentation but effectively just what's needed to plan and help people understand the code.
Having lived through both, modern development practice is superior in practically every way, most especially with respect to quality.
The problem with Heavy Development Methodologies is that they didn't actually work.
I have a blog post in the works about this but it's not ready to share; in short, I can't help but notice that hyper-focused, optimize-for-time-to-market, minimum-viable-product projects have a real, and frequently killer, problem once you have built your initial, hopefully-better mousetrap. Almost every company I've ever worked for has gotten that initial mousetrap done, tried to expand horizontally to actually have enough stuff-that-works-together to actually sell, and fallen on their faces because that initial super-specific development effort created not just code, but product assumptions that are prohibitive to unwind.
Most of them hit the hillside because reality is no longer playing ball with their (frequently VC-driven) needs to cut scope and ship.
What we should do is probably in your advanced data structure programming course that everyone has to take is to create a model of an elevator and diagram the behavior using a sequence diagrams. This would be achieved by using an associative array AKA a map [1] that would represent where the elevator is and what it has to do next (current state and next state based on input).
If you program this correctly it gets around using unit testing even because you have diagramed and all parts of the system are known and can just be gone through and that would be a sufficient test. An example of a library that implements this is at https://pypi.org/project/python-statemachine/
- There's very high-level engineering occurring ('computer science'). I always assumed this would be at places with 'web-scale' problems, but I've been seeing amazing work in local (not-web-scale) product companies here in Australia.
- On the other hand, I think a lot of other development work is not much above building flat-pack furniture. I guess this is where lo/no-code solutions will thrive
The important thing is to recognise which is being done, and which practices apply each case.
And thank goodness. I am all in favor of more consideration, more quality, more integrity. I like good architecture. But I think it's a giant mistake to create a software architect role. Every company where I saw that in practice, the average architect was a bloviator who did no actual work but was very excited to tell everybody else how to work.
At the code level, this resulted in a lot of messes. The edicts and white papers sounded good in theory to the kind of people who decided the bonuses of the architects. But frequently they were unworkable in practice, causing a lot of code that conformed to the theory but was a pain to actually deal with.
I agree with you that a lot of things are half-assed and rushed, of course. But we are never going back to a world of 18-36 month release cycles where people could stroke their chins for a quarter or two before building anything. Instead, we need to move in the direction of continually investing in quality, so that the design of systems improves over time. Something that I think is actually easier, in that waterfall design practices locked in important decisions early on, when people knew the least.
The project management triangle never stopped existing.
In fact, I believe it pretty much killed the profession of software architect. Many teams had it as a dedicated role, and this indeed would be a person documenting/designing systems using UML or otherwise. And they'd know the classics, like memorizing all design patterns. Finally, they'd use formalized architectural decision making methodologies to justify tech choices.
Nobody seems to do anything like that anymore. Everybody is half-assing design or skipping it entirely. Solutions are reinvented and tech choices made on a whim by the loudest person whom won't see the consequences of it anyway. Because we've told ourselves that shipping garbage in short cycles is the one and only way to do things.