I'm not the poster you're responding to, but for me in my career, I originally tried to design things in more detail up front and write a lot about how it would work.

After many years in the industry, I've actually found that it's better to quickly prototype the various "big pieces" of whatever it is you are designing and learn the constraints of the space you are in first. (You're never going to get everything right with that first attempt at a design.) You can write documents up front explaining what you think your code is going to do, but I would actually not recommend going into too much detail because you only really learn about the problem space you are in by doing.

In terms of a document, keep the initial design light and explain your assumptions, what the technical constraints you face are and how you plan to mitigate them. This is mostly to share out with other people to see if they have concerns about the approach and if they have ideas to help with the process.

+1 to prototyping. Having 100 lines of python that does 20% of what you want your full component should do is probably more useful than a 10000 word document outlining 95% of what your component should do. You can lie to yourself in writing but not in code.

That being said, documenting in horrendous detail your actual requirements and your assumptions about the state of the world is more useful than either of the above. When your PM says they want something in "real time" do they mean like RTOS, seconds delayed, or "twice daily" (since the old system was a monthly batch!)? Is it ok to assume the system is 99% correct instead of 100%? Is it ok if a user sometimes gets a duplicated notification?

These types of requirements and assumptions can affect the complexity of a project 10x if they are not handled properly.

Exactly. Prototyping reveals a lot of engineering constraints that you cannot see just by designing at a high-level. I've been bitten too many times by thinking something is simple because I designed things at a high level (without prototyping) only to find out there are hard constraints as soon as I start writing some lines of code.

I don't know if this is a common technique but I actually like to do a "big bang" type of prototyping where I build out the major pieces all at once to see what the general look and feel of the system is. From there I can make a way more informed decision on what I need to do versus something superficially high level.

For me, the most important thing from early designs is answering the question "what are the unknowns and what are the risks of this project?" and figuring out the right path to remove the unknowns and reduce the risks.

> You can lie to yourself in writing but not in code.

You can certainly fool yourself in code, and think you have the solution, or part of it, when, in fact, you do not. If this was not the case, prototyping would be meaningless, as you could only write correct solutions.

I would advise abhishekjha to try thinking one or two steps ahead before starting to write the corresponding code, and to pay particular attention to where it does not work out as expected, because without practice, one is unlikely to advance beyond trial-and-error programming.

This is a good point. Some of the techniques that I employ today (prototyping at a large scale) are ones that I can employ simply because I have experience and can see ahead to what potential problems there are in various designs. These things aren't obvious to a junior developer and letting them get too far with a prototype can be dangerous.

It's a lot of writing and rewriting. I usually start off putting everything I think I want in the spec into a text document without worrying too much about logical consistency, then leave it overnight, add more things, leave it overnight, reshape, etc. It's important to take a long break after hitting a stopping point so that you gain some perspective. It's also good to solicit comments from your network, although that tends to be as difficult as finding code reviewers, so YMMV.

You should also write a reference implementation, because actually implementing something will expose a lot of problems in your assumptions, and it will give other implementers something to test against.

Generally, it's 3-6 months from inception to release of a smaller spec if you want it to be solid.

Actually, you could follow along with a couple of specs I wrote since I have a git history of them:

https://github.com/kstenerud/streamux/commits/master?after=7...

https://github.com/kstenerud/concise-encoding/commits/master...

I think a lot of mature RFCs are based on prototypes. In my experience, prototyping is the first step in design; I know I've seen "designers" and "architects" hide that they prototyped something before writing the document they've been asked to. I know at least one did this to give the impression that they were able to express it fully-formed from whole cloth.

Sometimes you can get away with documenting a design based on prior art, but without creating your own prototypes.

For the most part, rather than the first, good design documents are the the second or later expression of a design.

Added: I think this ties into what jsty is saying. Good designs tend not to be the first iteration of an idea, regardless whether the iterations were intentional design, or refactoring.

IIRC, Joe Armstrong (RIP) was a firm believer in the "write it once, then throw it out and rewrite it at least once" pattern of development. I think that whether you're writing a design document or a new version of some portion of code, you are refining a design, and that tends to help.

Start off by writing down a clear goal of what you're trying to accomplish. This is pretty much the first thing I do when I write any document in fact.

Then figure out what the requirements are that help you meet your goal and write those down.

Once you have a goal and a set of requirements the design is usually pretty easy.