EDIT: Totally agree about the 'important mindest for startup', I had a similar eye-opening experience, working in a startup with 'cowboy' code that was actually quite good, but I had to unlearn a bit of stuff I read as a junior/mid-level developper.
It was code that was well-architected, had well-concieved data structure, and had business value, but every "professional" coder would deem bad-quality, since it had no unit test, did some stuff unconventionally, and probably would have many linter warning.
I drew some quick placeholder art for a game I'm working on for fun. One day I might sit down and draw some good art, but not today. When the day finally comes to finish the art, the difficulty of creating that art will be no greater than it is now. In fact, with improving art tools, it may be easier to do later.
On the other hand, if I'm not quite happy with my database scheme and I want to change it, I can spend a day on it now, or I can spend--based on my experience--months on it in the future. In fact, there's a good chance if I don't fix the database schema now, it will never be done.
But aren't unit tests the half-assed solution? They are a wonderful help if you want to deliver something fast, but if you had more time you'd probably do something else.
Going to mule that over.
Or, in this case, half your asspower.
I think the asspower unit of effort makes more sense than story points.
Meaning that e.g. if you really need a ride, then a noisy, beat up old car will do.
Plus a lot of our overcomplicated architectures on the frontend are because we shoot for a UI that competes with products that have been iterated on for 15 years now by teams of dozens or even hundreds.
There's overcomplicated, there's too basic, and then there's simple.
I think it's fine adding just enough complexity and abstraction to make something malleable and manageable in the future
Some of these projects were so late in their deadlines that had to be canceled
Speed is really important a lot more often than devs like to acknowledge when a company is small and fighting for its life to get revenue, let alone become profitable, and the code debt is often worth it. Fixing the code debt itself doesn't need to be any more thorough than necessary either.
This is something that I like about the metaphor of 'code debt', and which tends to go over peoples head: debt can be a perfectly fine instrument, and just like a real debt, it can be a good thing when leveraged wisely. The issue is more when debt is treated as 'free money', and is used carelessly.
Unfortunately people who create those monstrosities hide behind the same "keep it simple and quick" excuse than people who know how to prioritize.
It's interesting how many people here ignore that scenario - it's surprisingly common. Is it because most of developers jump ship at that point?
// really not the way to do this, but it works
// ideally, we'd do xyz, but this is good enough
It has become overwhelmingly obvious that the industry is never going to reach a state where technical debt and bad code is exceptional and high quality is the norm. If there is a maximum amount of pessimism I could have, it would be for this.
They'd rather be called "engineers."
With none of the licensure, mandatory education, and so forth. But the world needs programmers and technicians for most of the work we have to do. I do comparably little "engineering" and the little I have done that qualifies for such a statement I recall. Majority of the work is programming and technician work. Nothing wrong with that.
i.e. in Austria many IT techs and devs are called (at the doctor's office or in formal settings) "Mr. Engineer" ("Herr Ingenieur") if they fulfill some formal criteria and get licensed. A further qualification is becoming a federally certified civil technician for IT (Ziviltechniker or Ingenieurskonsulent) - also something that public sector contracts sometimes mandate.
Software around the world is engineered - it just often isn't in California.
Now, there are several types of taxes you pay in Germany based on your income. One of them is the income tax, the other is called Gewerbesteuer (maybe best translated as "commmercial tax) - and the Gewerbesteuer can be significant (something like 12% of your income). Many freelancers are exempt from the Gewerbesteuer, and so are software engineers. However, I learned that you only qualify as a software engineer if you have actually studied software engineering or a closely related field. I had studied business administration. I tried to explain to them that I have been working as a software developer for over 10 years, literally doing the same job as somebody who had studied software engineering - to no avail. They woulnd't exempt me from the Gewerbesteuer.
I'm not claiming that I'm an "engineer" or that my computer science fundamentals are as good as somebody who studied computer science, but the tax exemption shouldn't be on some bureaucratic difference like what you studied a decade ago, rather on what kind of job you're doing.
Welcome to Germany!
But I never really studied software engineering or anything related, only Electric Engineering.
But nobody in the government knows that. I don't have my diploma papers with me and have no time or interest in going to my country of origin and ask my university for it.
With a degree in computer science you are also allowed to call yourself an engineer in Germany, for anyone reading this and wondering.
As for formal education - I've seen dozens of people with an engineering diploma who barely can write a FizzBuzz, so it is not the best indicator and mostly people don't look at it when it comes to recruitment aside from big corps. Although that kinda changed lately as junior positions started evaporating and without experience the diploma is the main thing they check.
What that means in practice is that they study math and physics for the first 2-3 years of their degree, instead of computer science or software engineering.
Does that make them build better systems than Californian devs? Based on company revenues and salaries, I’d say no.
Majority of silicon valley devs are immigrants though, they are good because they are sourced and filtered from all over the world not thanks to American education.
Keep in mind that being the major economy also carries costs - the US spends many times as much as other countries on its military and has no realistic path towards spending less.
And either way, their economy has been wildly succesful since the 1800’s.
Can you explain the difference between 'engineered' software, and the rest? I mean other than being created by someone with an engineering certificate or whatever.
But, for me it implies that it gets Planned and Worked on formally like other engineering projects, Lists of Requierements, Detailed Propossals, Design documents, BOM's ,Gantt Charts, the whole shebang
The alternative being the ultra lean SV style, where a project is lucky if even the public facing documentation is not at least N-months out of date
Normally in startups it is very common to have the only available documentation be the code itself.
A gatekeeping organisation you have to pay to enter the workforce isn't the same as something actually being "engineered".
For something like this to work there needs to be a strong and technically competent government that the tax payers trust. This is presumably why it works in Austria and not the USA.
You can enter the workforce as a developer here without being licensed or credentialed at all, it's absolutely no problem (legally speaking) - it's just an obvious sign of a commitment to engineering responsibility and quality.
> they fulfill some formal criteria and get licensed
And just to troll a little bit: Do they make you do any hard problems from HackerRank or LeetCode?
This analogy falls apart the moment you apply it to anything in software.
Most of the time however stakeholders will prefer that someone unleash the last technical shiny tech stack (that no one master in the team and that is about to be dethroned by the even newest thing) to deliver a pixel perfect reproduction of the mock-up rather than question the relevance of the proposal.
Engineers shouldn't compromise on things like safety, security, legislation, budget, properly documenting, properly communicating with others, etc. Same as anyone working professionally with code should, no matter the title.
But there's nothing that says that engineers should only be doing groundbreaking or interesting work.
The "artist" analogy works much better.
EDIT: Perhaps the main difference is what psychoslave mentions below – engineers are expected to question the relevance and necessity of requirements, and work together with business, rather than just doing as asked.
He defines the Engineering Method as: 'Solving problems using rules of thumb that cause the best change in a poorly understood situation using available resources.'
In that sense a Software Engineer would not necessary need to know all the complexities of the complete system to make something work as specified. And I would equate a 'Programmer' more like the Scientist or Mathematician in this case.
Funny how there are so many different ways to look at a title.
I went to an engineering school. Actual engineers design bridges and machines and biochemical whatnot, and of course require licensing. I push code around.
I even try to subvert company culture and say things like "Well developers will need to update that API" instead of "engineers".
I would perhaps even like there to be a license, because then we would maybe get way fewer people interfering with a technically sound decision process. As it is right now, every manager dude thinks they know better what to build than the actual programmers/engineers, even if the programmers/engineers work on the product for years already and have gathered experiences the frequently rotated managers lack. It is always soooo convenient for the manager types to defer good enginnering endlessly, so that ultimately it never happens and shitty products are made.
https://en.wikipedia.org/wiki/Architecture_astronaut
...bikeshedding is when you have that all-day, all-teams meeting where marketing and design and management argue about whether to put X content in the header or footer.
The logical implication of technical debt having an interest rate (i.e. It costs more to fix something later than now) is that like money, features have time value. This is what makes it worth it to incur technical debt.
Thinking about the implied interest rate of technical debt makes it easier to rank what should be prioritized for a fix. High interest debt should be paid off first. But you might also focus on refactors that reduce the interest rate, such as by modularizing your code so it's easier to replace down the line.
If very likely, then you should invest in making the code high-quality. If unlikely, then you should half-ass the code strategically
I became very fond of #TODO ‘s.
# TODO: (CAFE-1256) Refactor steamed-liquid stuff to support non-dairy options
# We don't currently support non-dairy coffee additives, but once we do
# we will need to ensure that the steamed-liquid transport
# also alerts when users have a dairy allergy.
In the sense that the 'technical debt' is only something to worry about in the future, if your project goes anywhere. But if no-one ever uses your product, you don't need to fix the 'technical debt'.
I have no idea who said that, but I use it a lot at work when people want to cut corners with the intent to fix it later.
Did the business make money, though? I think that's the law of the jungle
Always write the simplest thing you can, but no simpler. Finding that line is where all the art is.
Maybe stavros' was hallucinating that strawman reduction in there, is what I think. Like you don't have to say it's a false dichotomy unless that's the only way you read it. The existence of something between the margins, should be obvious. Anyway, haha! :)
What is the difference between this microservice architecture that gets a lot of hate here?
Microservices are a lot more complicated. You'll need to manage images for each of the services, a fleet of servers on which the services will be deployed, an API for services to communicate together, etc. In many (most?) cases, a monolith architecture will be a lot simpler and work just fine. Once you reach the scale at which you'd actually benefit from a microservice architecture (most companies won't ever reach this scale), you can start hiring devops and other specialists to deal with the additional complexity.
What actually gets hate, I think, is not microservices themselves, but the fact that microservices are often used in contexts where they are completely unnecessary, purely because of big tech cargo culting.
> should have a virtual address space
It's easy to forget this complexity, but it comes at a cost in terms of performance and, above all, technical debt. The microservice will probably have to be rewritten in 5 years' time because the software stack will be obsolete. Whereas some unix utilities are over 40 years old.
Either microservice A calls into microservice B or there's a glue service calling into both A and B. Either way there's some deep API knowledge and serious development needed.
Compare with a (admittedly trivial, but just doing that is orders of magnitude less complex than web APIs) `ls -1 | grep foo | sed s/foo/bar/g`, exit codes, self-served (--help&al.) or pluggable (man) doc, and other "things are file-ish" aspects, signals (however annoying or broken they can be) and whatnot. There's a whole largely consistent operating system, not just in the purely software sense but in the "how to operate the thing" sense, that enables composition. The closest thing in http land would be REST, maybe, and even that is not quite there.
Because the Unix programs all use pipes as their interface. When you simplify and standardize the "API" composition becomes easy. Microservices are more like functions or modules each running as separate processes - if you use the same language for the services and glue you can just compile them all together and call it a program right?
https://en.wikipedia.org/wiki/Common_Object_Request_Broker_A...
This predated REST. And the rest is history.
Unix utilities are stand alone and used as needed for specific tasks. They hardly change much, have no data persistence, usually no config persistence other than runtime params, and don't know about or call each other directly.
Microservices are moving parts in a complex evolving machine with a higher purpose that can and do affect each other.
In practice, a way of splitting work up between teams. Also it makes it easier to figure out who to blame if something breaks. Also a convenient unit for managerial politics
So because manager X "owns" microservice Y, it's going to stay around so that they have something to manage. Over time the architecture mirrors the organization
This does not mean "it's a small component in a dedicated pipeline". It means "this is a component that's useful in many pipelines".
Small projects are easier to fix than large projects.
I don't think so. Time and time again the client will insist on stuff like "the customer only needs a single email address/phone number" but you're going to pay for that one later if you do the simple thing and add an "email" column.
Same for addresses.
And a whole bunch of other stuff...you need to normalize the heck out of your DB early on, even if you don't need it now. The code you can mostly make a mess of, just choose a good tech stack.
Sometimes messing up your fundamental architecture means that you hit a refactoring your company won't survive (while your competition grabs all the customers who wanted that feature your architecture doesn't allow).
This is where experienced lead engineers earn their worth - they understand which parts cannot be fudged and fixed later and need to be there from the get go.
You'll only pay if the project survives long enough for that new requirement to actually surface, which often it won't.
As much as I love simplicity, optimising for peak simplicity isn't always a good use of your time.
Simple enough is often good enough.
I can't even add a logger without having to refactor the whole 2000 line class while writing a whole-ass unit test for it. It's been a full year and I still have to deal with this sort of crap on a weekly basis.
The most ironic part is that most of the devs that cooked the spaghetti are now either in upper management or in the same devOps team that's demanding us to clean the mess up.
What you might do is write tests early, so at least you have integrated a test rig to build on if the system actually gets any use.
Most of the work on mature systems is like this, rewriting and refactoring stuff. If it is very concrete, non-abstract, it's generally easier to work with in this phase than if it was cleverly and abstractly architected. Even if it's a big spaghett.
Almost everything else about code is subjective, but value is objective.
As a struggling solo tech founder, I'm in awe.
By that point, the fast team would be on their third iteration.
Compare this to another colleague, with an almost exactly similar use case, who just downloaded an open-source LLM, wrote some glue code and set it loose in production. It's not pretty but it (mostly) gets the job done.
As the old addage goes "Perfect is the enemy of good".
Optimising for speed of execution only matters some times.
In my experience it's rare to correctly identify the bottleneck in code while writing it.
But sometimes when I really torture my brain I can spend a few days of doing mathematical proofs etc to come up with a even simpler solution.
That extra effort is only sometimes necessary. (But can be lots of fun to develop.)
Why? It is so easy, just think of the work being done and pick the big parts, those are the bottlenecks.
Only reason I can see anyone fail at that is that they don't know how long things take in their programming language, but that takes very little time to learn, and once learned you will know the bottlenecks before writing.
In so many cases the bottleneck is using bad data structures everywhere, that often gets you 100x runtime and doesn't show up in a profiler because it is spread out all over the codebase, that is the real bottleneck that never gets fixed and is why programs are slow today. To fix that you have to learn to know the bottlenecks as you write the program and not rely on a profiler. Profilers helps you find how long things take, they are really bad at helping you understand how to make the program fast.
Some view simplicity more as minimizing lines of code, e.g., less moving parts.
I view simplicity more as increasing lines of code, the goal being to make the code very verbose. Sometimes this means more moving parts, but smaller "movement" at each step.
There are other views of simplicity as well.
But sometimes you already have something that works, is correct and fast, but you still want to simplify: for example, when understanding _why_ that code is correct is too annoyingly complicated to explain and understand.
It’s factually correct due to hardware and compilers (néé transpilers) offering so much headroom, but part of me cries when you compare modern hardware utilization compared to how optimized-to-the-gills late generation PS2 games were.
I write a lot of data-pipeline code in Python that mostly runs once a day. Does it matter if it takes an hour to run instead of five minutes? Not really, as long as it's ready to go for the next run.
The issue is when you have people who do not code with efficiency in mind, and someone who does think about those things reviews the code.
Most efficiency gains are probably marginal and not that impactful. So you're probably OK ignoring it. And it's true that bringing such things up during code review and then going back and changing it will take more time.
But if people wrote the code with efficiency in mind to begin with they likely wouldn't be spending much more (if any) time while writing the code. Just have to use their brains a lil.
Literal example... replaced with a lock-free hashmap, and reduced the blink of an eye in terms of time.
1. Make it work
2. Make it right
3. Make it fast
A few years ago I worked on a system that had replaced a previous system that was like that: a bunch of microservices with multiple instances, communicating via message queues, all in the name of scalability and high availability.
The actual nonfunctional requirements? Handle between 0 and a few hundred requests per day, and it would be just fine of they get delayed a few days as well.
The best part was that the overengineered previous system actually had far more outages and delays caused by its complex deployment and data model than the simple single-instance monolith that replaced it.
JUST_DO_IT (add value, fast) <-----------------------------------> WHAT_THE_BOOK_SAYS (enterprise arch etc etc)
The BIG problem is the total misconception that:
> JUST_DO_IT = bad engineering
> WHAT_THE_BOOK_SAYS = good engineering
In reality, the quality of execution is independent of the development approach - you can make a complete mess of either approach, however, the WHAT_THE_BOOK_SAYS approach WILL cost you a LOT more time and money to discover your team has messed up it and WILL cost a lot more time and money to fix.
My experience is that, just as your product evolves & grows, your engineering strategy should evolve and grow. There's a good reason why fortune 500 companies will have enormous IT teams, using enterprise cloud technology... and there's a good reason why successful tech start-ups do the exact opposite.... but hopefully, the startup will become a fortune 500 company.
But at least in science roles it hasn’t happened yet. Rather, I keep seeing instances of bogus scientific conclusions which waste money for years before they are corrected.
Being systematic, reproducible, and thorough is difficult, but it’s the only way to do science.
Beyond that, my focus is on a solution to the problem/need at hand, and less about optimizations. You can avoid silly stuff, and keep things very simple from the start. Make stuff that's easy to replace, and you often won't ever need to do so.
Most software isn't about science, and isn't engineering... it's about solving real world problems, or creating digital/virtual solutions to what would otherwise be manual and labor-costly processes. You can shape a classic rdbms into many uses, it's less than ideal, but easy enough to understand. Very few jobs are concentrated on maximizing performance, or minimizing memory or compute overhead. Most development is line of business dev that gets deployed to massively overpowered machines or VMs.
However counterintuitively, I have stopped caring about waste, and have been more focused on the value. Waste you can always optimize later if you want to, value creation is the difficult part.
Instead it's combative code reviews where everything has to enter the code base perfect.
Makes me sick.
Optimizing for 1M concurrent users, paying down tech debt, refactoring, and testing are all things that engineers love to cite that will somehow make everything better. The customer won't feel a single one.
Spotify uses 42 different versions of jquery (among a dozen other libraries) and it's working just fine (with >1M concurrent users).
Even electron! That crap is so heavy, its (lack of) performance is notable. But! Many billion-dollar companies built on its back.
I laughed when an exec at Evernote told me that every year they go to MIT to try to recruit the top 1% of cs grads. What a waste of talent.
I've been working for the same company 10 years. When we built project A, I was surrounded by people who had built the same product for the past 20 years together. As a younger dev, I deferred to them as they were much older and had done this before. You would probably admire them. They pumped out code and delivered immense value. We had no over-engineering because there was no upfront design discussions. When we needed to use some service, we just wrote that directly in our normal framework code instead of using some over engineered abstraction. That's usually highly praised here on HN.
Years later, that project became an extreme maintenance burden. There is no separation of business code from framework code, so the first major framework migration took... years. When they completed the framework migration it was already 2 versions out of date. Subsequent upgrades were faster and given more priority, but still took multiple months. Onboarding new clients took whole teams. The code is fragile, lacks test, etc. The app is classic "spaghetti code". But they sure got to production quick!
Around 2.5 years ago, I was tasked with building project B. There was no development team. I worked with two domain experts, but ultimately the development of the product was in my hands. This is a product that will live on for at least another 10 years and will likely overtake our core product in revenue. There is no one to blame, but myself, if in 5-10 years this app becomes another unmaintainable mess. If you're in this position, what do you do? For me, all I could do is draw on my experience to try to avoid the mistakes I've seen made in the past. We made a lot of mistakes in project A, but I knew a big one was tying our business code, the core of our business, to the particular flavor of web framework that happened to be popular at the time the project started. How do you avoid that? Well, "Hexagonal architecture" sounds like a big scary buzzword, but it's a damn simple pattern that solves that elegantly. When we started, I basically received an indirect mandate that we use the same framework as project A. After we had written ~25k LOC, I had enough and we dropped that legacy framework and migrated to something much better. And we didn't have to change a single line of our business code. Our new framework allows us to iterate much faster than legacy framework. 2.5 years in and our code looks as clean as that initial POC I built. Hexagonal was a small bit of the initial architecture I did. I only point it out here as a counter example. But all the little things I've done which are considered "architecture" have paid off years down the road.
I'd like to think that, most probably, your job exists and is funded because those folks (and you) "got shit done" for the last 20 years.
In my experience (around 2 decades also), when it comes to small/medium projects, spending a bit of effort separating business logic from UI/database/server code is wise up to a certain point.
If the effort doesn't add too much code and layers of indirection, then fine.
However, DDD/HexagonalArch consultants tend to add soo much boilerplate and layers of indirection between between an HTTP request and a response that it becomes detrimental to productivity.
For example take a look at this article:
https://herbertograca.com/2017/11/16/explicit-architecture-0...
Specially this part (but the entire article is a gem):
> This means that Dependency Injection and Dependency Inversion are not enough to decouple components, we will need some sort of architectural constructs. We might need events, a shared kernel, eventual consistency, and even a discovery service!
Respectfully I'd like to think that it is satire. But it isn't.
With events your code is no longer linear. Stuff happens with triggers (events) that are not clear at first glance unless you fully understand the entire stack.
Shared kernel is just a place to shove interfaces under the carpet and pretend code is no longer coupled because you added extra 2 layers of indirection (interfaces and kernel) between 2 lines of code. Congrats now it became much harder to understand what actual code will be executed when I call this method.
Eventual consistency... Is that a good thing? We spent 4 decades maturing ACID compliant transactional RDBMS only for someone to come and disregard all these gains?
Discovery service... Yeah your code became so complex that you need more code to discover code. Because we can't have one layer calling another directly. Nonono, that would be coupling. And coupling makes it hard to change frameworks in a distant but slightly possible future. I'd rather pay a very expensive insurance from now on to safeguard from that possibility. Even though the insurance itself is so expensive that it alone can suffocate the project.
Sorry for the rant.
And that's where a balance is needed. No one seems capable of defining what good architecture is. We have plenty of posts about spaghetti code, architecture astronauts, etc. But there's few comments with success stories, especially about fixing a spaghetti codebase and what that project looks years down the road. And I think that's because the problem is multi-faceted.
It's not "just" a technical problem. From a technical side, it's primarily lots of little patterns that as a whole add up to producing quality code. But there's also the social side, to hiring, reining in devs, enforcing the code patterns and quality, pushing back on business, etc. I can give lots of examples, but this would be a long post. All of these factors ultimately determine the fate of the project, the quality of the code, and whether it becomes spaghetti code or an over-engineered mess.
The code is not brilliant, it is full of technical debt (as thats what happens when companies grow and change) but is still perfectly maintainable.
I have been watching the herculean effort of three different teams to replace this system, carving off the low hanging fruit to NetSuite and React/Java applications while repeatedly delaying the core system while they agree on standards to use.
I stick around as I am quite close to retirement now and the employees and the company are good to work with and don't deserve me to abandon them. At this point I genuinely want the projects to succeed so I can move on as there are many small companies still out there that I can make a difference in even with my age, single man risk and obsolete software.
I've seen entire teams burn so much money by overcomplicating projects. Bikesheding about how to implement DDD, Hexagonal Architecture, design patterns, complex queues that would maybe one day be required if the company scaled 1000x, unnecessary eventual consistency that required so much machinery and man hours to keep data integrity under control. Some of these projects were so late in their deadlines that had to be cancelled.
And then I've seen one man projects copy pasting spaghetti code around like there's no tomorrow that had a working system within 1/10th of the budget.
Now I admire those who can just produce value without worrying too much about what's under the hood. Very important mindset for most startups. And a very humbling realization.