I think one of the causes is that a lot of CS is taught top-down and not bottom-up, so things like flowcharts, state diagrams, and other similar visual aids are completely skipped in preference to more abstract thinking. I wonder if at least some of this aversion is directly related to the "gotophobia" that started with Dijkstra's famous paper.
It may seem difficult and tedious, but this is how you engineer something rather than merely 'develop' it.
As someone who learned the "old school" way of planning out your code --- yes, even drawing flowcharts and state diagrams --- before writing a single line of it, I can assuredly say that it should be a natural and completely pleasant process; what's "difficult and tedious" is trying to debug and coerce something you (or someone else) wrote without planning into working correctly.
Over the years coworkers have been puzzled at why I spend "so much time" flowcharting and just thinking before writing even a single line of code --- only to try adopting the same method once they realise how much time it saves in total, since all the cases have already been accounted for and the only thing left to do is test out any of the few remaining and often trivial bugs introduced during the design->code process.
The developers insisted on all behaviour being specified and documented before they would start coding. Of course, a truly exacting specification is equivalent to code. So the users who had been tasked with describing the required functionality ended up inventing their own DSL and “coding” the entire system in Microsoft Word.
Needless to say, they were the only ones who understood the spec. IT cried foul. Business cried “this is what you asked for!”
Cue months of arguments over what constituted a proper spec, during which the users essentially taught themselves how to code. The software eventually ended up being written by transliterating the DSL into actual code.
It worked, but remains to this day an unmaintainable mess.
Essentially the same problem, just on a higher level of abstraction.
Oh, and failing to document DSLs is really a common mistake: No docs, no spec, no helpful error messages, and so on. Just like the very early parsers or compilers. But DSLs are really the most important thing to document, as these are so much harder to get into ... the "simpler" parts of the program are usually readable and understandable with less effort, even with pure code and no docs at all.
What's wrong with demanding to document a DSL?
That is, the DSL as a language is still much simpler (fewer data types, fewer elements, fewer rules, etc.) than the whole application, which is built using that DSL.
So I'd expect the DSL's documentation to be short and targeted at the people writing within that DSL.
If the DSL's documentation is more complicated than the original problem (spec), it failed at its purpose.
BTW, ultimately you have a chain of languages, such as:
Spec <- DSL <- Meta-DSL <- ... <- programing language
But in the end, that chain goes from complex/specific to simpler/generic. So the documentation effort should shrink dramatically from one step to another (or your system design is seriously flawed). Usually, the Meta-DSL or Meta-Meta-DSL is already the plain, generic programming language you are using. And that one requires no documentation writing at all, because is already documented and widely understood.Congratulations!
You see that "X hours ago" to the right of the username? Yeah, click that...
But it's a good tip nonetheless. Once you click the "x minutes/hours/days ago" link, besides bookmarking the resulting URL in your browser or elsewhere, you also have a few more options that show up at the top of the comment page. One of them is to "favorite" the comment. You can also favorite a submission in a similar way.
Just be advised that your favorite submissions and comments lists are public.
I totally agree that this is an example of insane, hidden functionality. How could anyone know that "click timestamp" = "view details"? Makes no sense.
However, seemingly every site/app has adopted this, from Twitter to Facebook to several PM services.
Why not acknowledge that this is a problem and just add a little "Permalink" link, which some sites used to have? Or at least a permalink icon?
Surely the combined design talent of Google, Facebook, and Twitter can solve this problem better.
A hypertext system that doesn't treat links as slightly opaque, explaining them in detail wherever they appear, sounds unpleasant to me.
I guess you could probably modify most such systems to have a flashing popup on every link that said "There is more information available if you click on the text here!!!!".
The question is: is it a worse mess than if you'd have started without the spec? My intuition says "no", and the experience the users gained regarding formal thinking is a huge net benefit that will be an asset for years to come.
So, take this with a grain of salt.
I can't code without a diagram. I make a diagram and process the logic. I make lots of sub diagrams. I make notes and ask myself a lot of questions. I'll also ask other people, if I can.
I can't do it. I can't program 'on the fly.' My attempts have resulted in horrible results, not even qualified to be called results. If I don't process the logic, pretty much from start to finish, I can't do it. I use a notation system that isn't even a real programming language and leave lots of room to cram more stuff in and make corrections.
Most of this is past tense, I don't do much anymore. I'm retired and my efforts are just little automation tasks for my own needs.
Anyhow, I figured I'd offer another perspective/opinion. I envy you if you can do it off the cuff. I've tried, I can't.
I have coded on-the-fly and I have coded from diagram first approaches.
on-the-fly works for me when there is an obvious data structure to work to, otherwise it takes a lot longer than planning it out because of all the edge cases
So my tip here is if you want to learn how to do it on the fly you should start with small steps by reducing the amount of stuff you need to see in a paper before start coding. In the other hand it's perfectly normal and there is no problem in doing what you do.
This problem goes away once you gain some experience - both in writing specs and in programming.
Think of it this way: do engineers build bridges without design documents and diagrams? No. So why are you writing software without them? If you aren't familiar with the problem domain, fine, do it the agile way. But once you gain experience there is no excuse to not thinking problems through in advance before trying to solve them.
Documents are cheaper to create and easier to change than bridges. Code is just another kind of document.
I've a sneaking suspicion that bridge building would look more like software if we could drive over AutoCAD models. (Or that software would look more like bridge building if we were still compiling by hand).
EDIT: I should add that I certainly do write and communicate documents about "what business problem does this even solve" and "what does it do" and "what are the major components and what are they responsible for" and "did you consider X alternative design" but planning down to the level of function signatures has usually turned out to be a waste of time.
That really depends on the type of code you are writting.
Writing code going into a space craft, or Nuclear Power Plant, or Encryption code that can be relied on by people to protect their actual life, or code going into a medical device or.... you get the picture
Yes if your writing code going into the next Pokemon time waste game sure you can be fast and lose, but to not presume all code is the same.
In this instance, code is going into devices that historically do not get updated properly, I think that warrants taking a more "bridge building" approach and less "Code is just a document" approach
The more important code is, the more I want to see it subjected to a realistic and comprehensive test suite, careful peer review, etc. Although you're right, I ultimately work on backend systems that can cause some downtime at worst, not industrial process control or crypto libraries. I also have the luxury of doing continuous delivery, incremental rollouts, feature flags, staging environments with shadow traffic, etc. so there are a lot of safeguards standing between a mistake and widespread impact.
Why on earth would you want bridge construction to be more like software engineering? Bridges work. We know how to build them bug free, every time. We don't know how to make working, bug-free software. There's even an old, not very funny joke about exactly this comparison. (Spoiler: it doesn't flatter the code monkeys.)
You might want to look up the Cynefin model[1]. Bridges are complicated, software is complex. This supports your bent towards experimentation, sort of; when solutions are unknown, that's required. But documentation and planning are part of what turns complex in to complicated, and that's a big part of the trick to successfully delivering big chunks of relatively bug-free code.
We don't build our "bridges", our compilers/interpreters do.
Also, flowcharts are less readable than most programming languages in non-trivial cases. Graphical programming languages are mostly useful as DSLs. Text is unbeatable in the general case.
As will the need for planning beforehand.
I enjoy both diagramming and early exploratory coding.
I have real problems when one of my (really smart) colleagues wants to discuss some code we have neither written nor diagrammed yet.
For her however that seems to be the natural way of doing it and when we started working together I had to point out that just didn't work for me.
It seems to me that knowing and being aware and accepting of the different learning styles can solve a number of problems in teams.
In other words, how would you express in code: "state Z can only be entered through either state W or X, but not through state Y", or "state V must be followed by state W, but only after condition A"? Sure, you can write a switch statement. But half the rules of a state machine can only be implemented by omission (i.e. by not coding it, instead of hardcoding the requirements).
But then the changes start and now I either discard the diagrams or accept that I have two things to maintain. Generally I take the former path because divergence from the plan is almost inevitable on anything that's alive.
For some people (like me) thinking through problems with pen and paper is a tool.
For others I guess it is annoying: look at him, now he's doing this wannabe software architect stuff again.
Same might go the other way: here he goes again, trying to prove his programming super skills by jumping into it without even the smalles amount of planning.
Seemed relevant to point out that at least for me it’s not.
For me it is not very visual when I code. It is more a gut feeling: this goes here, that goes there. This is good, bad, weird but works etc.
This is nice for me but I know I have to work on how to explain it to others.
Bad example but I guess this is somewhat closer to what I want to be able to explain: "it currently works but it is a bad idea because it means you'll have to remember to patch seemingly unrelated functions x, y and z whenever you update this.
Also, in this case it is even worse as in the worst case you won't see the errors until the quarterly reporting run and that code is well known for having bad test coverage."
Edit: as usual I'm curious as to why parent was downvoted?
Here is what the paper actually says:
The 802.11i amendment does not contain a formal state machine describing how the supplicant must implement the 4-way handshake. ... Fortunately, 802.11r ... does provide a detailed state machine of the supplicant.
The problem, as I understand it, was that when the protocol was formally verified, the properties checked were only the escape of the private key and identity related issues. The property of the nonce never being reset was simply not considered.
The missing piece, then, wasn't a precise description of how the system works, but rather a precise list of what it's supposed to guarantee. Without a clear and concise description of the requirements, it's hard to verify that the system satisfies them. Alternatively, it's possible that the list of guarantees was complete, in which case reviewers should have noticed that no nonce reset isn't one of them, yet it's required for the security of certain communication protocols used over the channel.
I shared the LTSA model with a few others at the time, but no-one else seemed very interested in doing anything with it.
https://www.doc.ic.ac.uk/ltsa/
https://tools.ietf.org/html/rfc4340
Edit: I dug through my old backups, and I think this is probably the model I used:
http://nrg.cs.ucl.ac.uk/mjh/dccp.lts
It's nothing fancy, but my point is that even simple modelling can spot errors that a whole standards group of smart people miss.
That must be the origin of all the useful but unofficial SVG ones I'd seen :)
Beurdouche et al., "A Messy State of the Union: Taming the Composite State Machines of TLS"
From the abstract: "mplementations of the Transport Layer Security (TLS) protocol must handle a variety of protocol versions and extensions, authentication modes and key exchange methods, where each combination may prescribe a different message sequence between the client and the server. We address the problem of designing a robust composite state machine that can correctly multiplex between these different protocol modes. We systematically test popular open-source TLS implementations for state machine bugs and discover several critical security vulnerabilities that have lain hidden in these libraries for years ..."
Based on this work you might say that there are formal state machines for TLS, but they aren't in the standard.
Let's reflect: iterative design is popular in software engineering because software usually suits it. Changes are easily iterable, you can often roll changes back, there's relatively low cost to cloning changes across millions of instances, code's design lifetime is relatively short, etc. In contrast, 'real world' engineering (civil, heavy mechanical) favours FEED because the cost of change rapidly and prohibitively rises with time, so nailing down the scope and structure early is critical, there's long procurement times for changes, the design lifetime of the finished product is measured in decades, etc.
The rub is that there's a small niche of software design, such as protocol design like this, or certain firmware scenarios, which far more closely resembles the latter even though it's usually done by a community used to and trained in the former. Although it's for different reasons, all the things that make FEED appropriate for 'real world' engineering also apply to these sorts of software tasks, but software engineering as a whole doesn't appear to[2] focus on teaching and supporting the necessary tools and techniques for performing that process well.
Flip that on its head, and it's why software companies quickly begin to run rings around companies traditionally focused on hardware once software starts to play a bigger part and the limitations stopping iterative design from working begin to disappear. Software engineering largely biases towards iteration (understandably) and when it works it allows for good advantages. But in embracing those advantages, software engineering as a discipline has begun to lose an understanding of FEED and the careful methodology it delivers.
[1]: https://en.wikipedia.org/wiki/Front-end_engineering & https://en.wikipedia.org/wiki/Front-end_loading
[2] I admit I've not done a CS/SE degree, but I've gone through a couple of subjects run online and I talk to friends who did do it and that's the feel I get. Besides, just look at HN's generally trending articles.
edit: With all I wrote, I don't mean to undermine the extremely good work that the devs designing these protocols, etc. almost always perform. It's far beyond my ability. I am speaking more in a general sense about how Soft. Eng. seems to have thrown out the baby with the bathwater in declaring agile philosophies as 'the best way'. My feel is that the pendulum seems to have swung too far past 'best' towards becoming 'only'.
Don't agile approaches encourage this, focusing on features and things that can cleanly be assigned story points at the expense of a bigger picture, with velocity being the focus?
If the spec is not wrong, careful planning is probably better than high velocity iteration.
I think cryptography is a good example of this--would you develop a cryptographic API on which people's lives might depend iteratively through a release early and fail often method? I hope not.
In a lot of software development, especially large scale enterprise software (where you're trying to map a complex set of existing business practices and workflows into software) and embedded software (where you have to write something that provides fixed, well-understood functionality running on hardware with a months- or years-long iteration time) you do have a complete and correct spec (or at least the means to generate one). In these cases a less ad-hoc approach than Agile can be beneficial.
Totally right.
Additionally, in my years of working as a contractor, I can tell you that nearly 1 in 1,000 companies (+ or - 1) care about security. Management simply does not value it, and everyone already thinks your product "should have been done yesterday." The entire world runs on insecure products. From banks, to healthcare, to universities. I can list every one of those who has put admin credentials in a publicly, internet accessible text file. (But they'll do silly things like insanely unmemorizable passwords like 23ix2n5 that get written down as postit notes and as files on desktops, and disable RDP copy-and-paste but leave RDP hard drive support.)
We're going to continue seeing these blunders day after day until people start going to jail for losing data.
I guess this is fine for throwaway acquihire SaaS stuff, but it's becoming a serious problem for stuff that's meant to be widely used.
I'll wait.
This, to me, is one of the biggest problems facing our industry. In first year undergraduate in engineering we learn how to design a state machine. We learn to design one before we learned OOP and circuits and assembly. Yet, how many APIs or Protocols have you come across or has your organization developed where there isn't a state machine described somewhere. It may seem difficult and tedious, but this is how you engineer something rather than merely 'develop' it. We programmers sometimes spend far too much time juggling stuff in our heads and communicating with coworkers in a wonderful game of workplace telephone. If the state machine has become too complex, then we have lost all hope to fully manage it and must accept the risks. Some things just shouldn't be risked!