News for hackers. Stuff that matters.
Re:Technical Debt: Why it'll ruin your software by solomatov on Friday October 23, 2020 @01:23PM (#24871760)
In my experience, I have seen much more problems because of over engineering by well meaning people than because of technical debt.

Simple but under engineered systems are much easier to rewrite than to simplify over engineered ones.

Re:Technical Debt: Why it'll ruin your software by spamizbad on Friday October 23, 2020 @01:37PM (#24871883)
One thing I've learned is that what often looks like overengineering is actually underengineering -- it's much "easier" to throw together a complicated system than really think things through and develop something that accomplishes the same objectives in a simpler (but possibly more complex) manner.
Re:Technical Debt: Why it'll ruin your software by hinkley on Friday October 23, 2020 @02:51PM (#24872573)
A blogger I read when people still did such things asserted that this sort of overengineering is fear. It's really dreadfully obvious once you know to look for it.

People are so afraid of making a bad decision that they refuse to make any decision at all, and then make a giant mess in the process. What you should do is spend your energy on finding reversible decisions, and then not spend a lot of effort on actually making them. We know where the paint store is, we know they can make up paint in 15 minutes, fuck it, paint it blue, we can always paint over it later. Hard no to black, though, since you can't paint over that shit.

People are so used to avoiding decisions that on a few occasions I've entirely flustered someone who wanted to tear into me (sometimes with an audience) but cutting them off and saying, "Yeah that was a mistake, and here's how we're going to fix it." None of them had any idea how to recover from someone saying "I was wrong" and going on to try to fix the problem. I still have a little video in my head of one guy's eyes bugging out when he realized what I just said.

Re:Technical Debt: Why it'll ruin your software by lt on Friday October 23, 2020 @02:22PM (#24872300)

    “I apologize for such a long letter - I didn't have time to write a short one.”
― Mark Twain
Re:Technical Debt: Why it'll ruin your software by mixmastamyk on Monday October 26, 2020 @02:14PM (#24899218)
I've seen that attributed first to Pascal:

https://quoteinvestigator.com/2012/04/28/shorter-letter/

There may be better information.

Re:Technical Debt: Why it'll ruin your software by coddle-hark on Friday October 23, 2020 @02:05PM (#24872157)
I think poorly engineered is a better term than under engineered in this case, but I agree with you.

Logging is my go-to example for this sort of behaviour. People write stupid amounts of log statements in their code, so much that it’s hard to even read the code to understand what it does, in the hopes that it’ll make debugging easier. Use a damned debugger! Take a traffic dump! Use strace!

What’s more, it’s extremely rare that libraries provide logs themselves. So the actual complex parts of your application, like say the HTTP library or (God forbid) the TCP stack, can’t be debugged this way.

If you find yourself writing a bunch of statements like “DEBUG: updating balance from 1 to 2”, stop and write some tests instead.

Re:Technical Debt: Why it'll ruin your software by jcelerier on Friday October 23, 2020 @02:30PM (#24872384)
> Use a damned debugger! Take a traffic dump! Use strace!

I have encountered a fair amount of cases where that does not work.

- debugging a super large program ? loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds.

- not an admin on the machine you are and the person with the admin account is not around ? sorry, you can't debug on macos (and likely in some linux distros)

- likewise, no traffic dump (and I'd assume no strace) if you don't have root access

Re:Technical Debt: Why it'll ruin your software by SCHiM on Friday October 23, 2020 @02:34PM (#24872429)
And how about highly congested multi threaded programs?

I've got a program with log statements like that all over the place. Since stepping through it with a debugger would not even be possible. My IDE takes care of hiding the debug statements, since they're all encapsulated in different regions and if-def statements to log different types of things.

Re:Technical Debt: Why it'll ruin your software by coddle-hark on Friday October 23, 2020 @02:56PM (#24872633)
Isn’t that what tracepoints are for though?

Flipping your argument, you’re basically saying that you can’t debug multithreaded code if the program doesn’t log.

Re:Technical Debt: Why it'll ruin your software by SCHiM on Friday October 23, 2020 @03:14PM (#24872797)
I certainly wouldn't want to debug this ball of mud if there were no log statements, no :)

But, I think we're depending on different programs in our workflow. I much prefer a logfile with statements that focus on what I want to inspect at the time.

I'm quite proficient with debuggers on both Windows and Linux, but I tend to use them less when dealing with my own multi-threaded code.

Re:Technical Debt: Why it'll ruin your software by merb on Friday October 23, 2020 @06:53PM (#24875206)
that works maybe with multi threaded code but not with greenthreads that are all over the place. the calls stacks are a pain in the ass.
Re:Technical Debt: Why it'll ruin your software by coddle-hark on Friday October 23, 2020 @02:51PM (#24872574)
> loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds

To be clear I’m not talking about adding some specific print statements to the code while debugging, I’m talking about keeping those statements in production code.

Of course, there’s cases where logging might be your only option. In that case: Go for it!

Re:Technical Debt: Why it'll ruin your software by MaulingMonkey on Friday October 23, 2020 @08:21PM (#24875797)
> - debugging a super large program ? loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds.

Relinking a super large program may take minutes, whereas loading gdb can get me a backtrace in 3-4 seconds! VS natvis files and other visualizers can hot-reload while I'm looking at a crash dump from production that took hours to trigger/gather/repro!

There are cases for logs - where sufficiently efficient conditional breakpoints are particularly painful to setup, for example - but the only time I've waited minutes for a debugger to respond has been with VS after major system updates as it refreshes a universe of symbols from the symbol server.

Do you have some really slow python scripts auto-loading in GDB or something?

Re:Technical Debt: Why it'll ruin your software by jcelerier on Saturday October 24, 2020 @07:44AM (#24878574)
> Relinking a super large program may take minutes, whereas loading gdb can get me a backtrace in 3-4 seconds!

well, we have definitely opposite experiences :) the main software I work on creates a ~1gb binary in debug mode and lld links that in a few seconds. Bud gdb and lldb, even with gdb-index, and all the optimizations I could find, both make startup slow enough that I can go for a coffee and it's not always finished loading when I'm back

Re:Technical Debt: Why it'll ruin your software by nullsense on Friday October 23, 2020 @06:17PM (#24874881)
On the contrary, I've spent months recently working on a system loaded with tech debt that has fallen victim to a half done rewrite. I've used a debugger on it plenty of times.

It was only when I added proper logging to the critical junctures in the code that I could finally see very clearly how it behaves and see what input results in what code paths and output. I learned more in an afternoon than I did in a month.

Re:Technical Debt: Why it'll ruin your software by m463 on Friday October 23, 2020 @01:47PM (#24871992)
Sometimes it's hard to tell the difference.

for instance one smart and experienced guy I knew said if there was an error, your program should just fail instead of giving a ton of messages and recovering.

and if you're just throwing code together, that might be what you do.

on the other end of the scale, a well written "second pass through everything with cleanup" might have tastefully written code, a few relevant comments and consistency.

and then I've seen a lot of code that appears to look like that, but is copy/paste garbage. (sort of the coding equivalent of a wordpress theme)

Re:Technical Debt: Why it'll ruin your software by pjc50 on Friday October 23, 2020 @02:59PM (#24872659)
I've come round to the "fail hard and early" view, provided you can be sure in your design that it doesn't leave junk lying around. Because otherwise contaminated data and invalid states start propagating everywhere..
Re:Technical Debt: Why it'll ruin your software by quicklyfrozen on Friday October 23, 2020 @05:14PM (#24874260)
This can be fun if it's a particular client request that causes a hard failure, and the client retries several times before giving up. Before you know it, all your instances are restarting and no traffic is being handled...

So fail early, but not too hard.

Re:Technical Debt: Why it'll ruin your software by ziml77 on Saturday October 24, 2020 @12:35AM (#24876971)
Interestingly I prefer failing hard specifically for data consistency reasons. I've had to deal with systems that swallowed far too many errors and let themselves get into an inconsistent state which then caused other operations to not fail but instead persist invalid data. It's very hard to clean messes like that up (restoring a nightly backup is easy, but that's a last resort).
Re:Technical Debt: Why it'll ruin your software by darkerside on Friday October 23, 2020 @02:28PM (#24872363)
> your program should just fail instead of giving a ton of messages and recovering

I would bet that he was right in that situation, but that he was also making a nuanced statement. Not all code should fail loudly, but when it should, "recovering" can sometimes be a distraction from a very real problem that needs to be addressed by the right person.

Re:Technical Debt: Why it'll ruin your software by flyrain on Friday October 23, 2020 @01:49PM (#24872009)
Totally agree. That complicated system is called a big ball of mud. Pretty easy to make.
Re:Technical Debt: Why it'll ruin your software by allenu on Friday October 23, 2020 @01:52PM (#24872042)
Yeah, to me there's a natural progression in understanding of system design.

When you start out, you just write code and don't think about how things interact with each other too much.

Then you learn that you're rewriting a lot of code, you overcompensate and start to over-engineer to avoid duplication and often to handle more scenarios than you need. You abstract out possibly too much.

Finally, you get to a phase where you realize a lot of abstractions that you've created aren't actually used and you're writing more "meta code" than you need. In this phase you learn to engineer just enough for what you need at the time and design things to be easily changed in the future. You trust that even if you're system design can't handle everything right now, you've designed options for yourself to expand as needed.

Re:Technical Debt: Why it'll ruin your software by Nimitz14 on Friday October 23, 2020 @02:07PM (#24872175)
Nowadays every CS student reads HN and blogposts like this one so by the time you end up having to work with them they're so indoctrinated into the "must write Enterprise(tm) ready code" mindset trying to teach them to be more relaxed is a real pain.
Re:Technical Debt: Why it'll ruin your software by dstick on Friday October 23, 2020 @06:06PM (#24874801)
Yup, adopt to the business you’re in. As in, the company. Make suggestions if you think it’s worthwhile but in the end accept that there’s a “that is the way” in any company and you’re getting paid to learn and work with that “way”, and build it out :)
Re:Technical Debt: Why it'll ruin your software by adkaplan on Friday October 23, 2020 @04:13PM (#24873509)
this last phase in your coder evolution reminds me of a very rough translation of the wabi sabi view on manufacturing. "Nothing lasts, nothing is perfect, nothing is finished."
Re:Technical Debt: Why it'll ruin your software by nickbauman on Friday October 23, 2020 @02:19PM (#24872282)
TDD helps keep this "abstraction abstraction" thing in check.
Re:Technical Debt: Why it'll ruin your software by mjcohen on Friday October 23, 2020 @03:05PM (#24872717)
I prefer TDD. More entertaining.
Re:Technical Debt: Why it'll ruin your software by spaetzleesser on Friday October 23, 2020 @01:30PM (#24871824)
I think a lot of today's massively tested, microservice and dependency injected systems will end up as huge piles of technical debt in the future. If you think modifying a COBOL codebase today is hard, wait another 20 years and see how much "fun" it will be to rework today's systems once their platforms are viewed as outdated.
Re:Technical Debt: Why it'll ruin your software by foobarian on Friday October 23, 2020 @01:51PM (#24872029)
I'm sure you would love working with currency conversion in a Kubernetes operator :)
Re:Technical Debt: Why it'll ruin your software by didibus on Friday October 23, 2020 @01:29PM (#24871810)
I would totally count over engineered a technical dept. Any design choices and implementation choices that make a code base limited in its future abilities to change and evolve, and which causes it to require more time and effort to be understood, I consider technical dept.

The dept is that you take time and effort from your future to save time and effort in your present.

Re:Technical Debt: Why it'll ruin your software by solomatov on Friday October 23, 2020 @05:27PM (#24874385)
I would call it bad investment not a technical debt. As an investor you could invest in a lot of businesses which fail if we use finance metaphor.
Re:Technical Debt: Why it'll ruin your software by mrmrcoleman on Friday October 23, 2020 @01:30PM (#24871819)
Agreed. I once worked on a project which was a microservices based system for managing various types of infrastructure. Progress was painfully slow.

There was a separate testing client that was used for a lot of the demos. The demos on the 'real system' would fail so often that somebody came up with the idea to just take a lot of the code from the microservices and drop it straight alongside the testing client, as a monolith.

It was so much more stable that after becoming the default way to demo, it became the default way to use the tool.

5 years on and I believe it's still running that way. In this case microservices were massive over engineering.

Re:Technical Debt: Why it'll ruin your software by 35fbe7d3d5b9 on Friday October 23, 2020 @02:32PM (#24872413)
> In this case microservices were massive over engineering.

They often are.

Well-meaning engineers and architects seem to reach for them while forgetting that microservices are about scaling your organization, not your software.

There are very few technical deficiencies in a monolith that are solved by microservices. Indeed, they typically bring technical hurdles: distributed systems are hard to reason through and IPC over the network introduces latency. And you need a very strong ops team to implement them without major headaches.

But where they really start to shine is when regression testing takes ~days and your development cycles are screeching to a halt because you've got too many in flight features and not enough runway to land them.

What takes real skill is recognizing which of your monoliths should be grown and which ones need to be sliced down the middle.

Re:Technical Debt: Why it'll ruin your software by jillesvangurp on Friday October 23, 2020 @02:27PM (#24872356)
Under and over engineering are relative to your understanding of future requirements, which by definition are both unknown and subject to educated guesses. Making such guesses correctly is what distinguishes a good engineer from a bad one. Nobody ever spells out the current requirements to any level of detail even; so forget about knowing about future ones. You just have to know, recognize, and adapt to that constantly.

A lot of over engineering is due to getting this completely wrong: you prepare for something that never happens, at great cost and while underestimating the difficulty level. So you end up wasting endless amounts of time on stuff that has no business value.

On the other hand, a lot of under-engineering is due to not having a clue about what is obviously coming next and getting caught by surprise by completely obvious requirements.

Re:Technical Debt: Why it'll ruin your software by karmakaze on Friday October 23, 2020 @04:28PM (#24873721)
> understanding of future requirements, which by definition are both unknown and subject to educated guesses. Making such guesses correctly is what distinguishes a good engineer from a bad one

I would rephrase this as make guesses based on probabilities and urgency estimates from past experience. You can't know the future but you have stats of the past. So a good engineer is one that has relevant experience and applies it appropriately.

Re:Technical Debt: Why it'll ruin your software by nullsense on Friday October 23, 2020 @06:25PM (#24874961)
Basically a perfect description of the problem.

Seems to me the default is obviously getting things wrong whether that be over or under engineering.

How would one go about reliably making choices that strike exactly the right balance consistently?

One thing I'm starting to really internalize is that to do so requires a deep understanding of software engineering as a domain, the domain and existing system, and the technical vectors in the business.

Re:Technical Debt: Why it'll ruin your software by miltondts on Friday October 23, 2020 @01:57PM (#24872078)
It's kind of amazing that "over engineering" in software means the exact opposite of what the term should mean.

To me, engineering is taking a problem and the corresponding constraints and building the solution that satisfies it with the least amount of resources.

"over engineering" in software is about trying to make something adaptable? Or maybe just satisfying someones sense of beauty.. Or maybe it's just about programmer convenience at the expense of customers/user experience and hardware resources..

Re:Technical Debt: Why it'll ruin your software by ironmagma on Friday October 23, 2020 @02:15PM (#24872248)
It's also a bit of an abuse of the term. Overengineering in engineering (mechanical, civil and other similar disciplines) means building a component to withstand more than the absolute minimum. So, a bridge you'd want to be overengineered by a factor of 2 or 3 at least, whereas airplanes are hardly overengineered (lower factor of safety).
Re:Technical Debt: Why it'll ruin your software by Ma8ee on Friday October 23, 2020 @04:13PM (#24873518)
But isn’t that how the term is used? The system is over engineered when it is built to scale to a million concurrent users while we current only have a hundred. Or when it is built to handle use cases that no one has required.
Re:Technical Debt: Why it'll ruin your software by ironmagma on Friday October 23, 2020 @05:46PM (#24874595)
In this context I read it more as being excessively complicated to some unspecified end rather than engineered to withstand a load way larger than necessary.
Re:Technical Debt: Why it'll ruin your software by Ma8ee on Saturday October 24, 2020 @04:09PM (#24881638)
Often it is excessively complicated because it’s divided into dozens of services to be able to scale, or having extra abstraction layers to deal with imagined future requirements.
Re:Technical Debt: Why it'll ruin your software by ironmagma on Saturday October 24, 2020 @11:18PM (#24883609)
Sure, but the fact that those requirements are imaginary instead of real is why it’s not actual overengineering. Overengineering is done with a specific dimension of scalability in mind for something that probably will happen, and there’s a budget for it too, etc.
Re:Technical Debt: Why it'll ruin your software by miltondts on Friday October 23, 2020 @02:33PM (#24872418)
Ah I was not aware of the general term and its' meaning. As you say, quite the abuse of the expected meaning.
Re:Technical Debt: Why it'll ruin your software by darkerside on Friday October 23, 2020 @02:34PM (#24872428)
I use the term to describe the introduction of unnecessary abstractions in the name of hypothetical future use cases. For example, using an enumerated collection of values where a boolean would suffice.

It could also refer to building resilience to load before it is necessary. Example, distributed databases with failover and recovery capabilities for a side project.

Poke holes all you want, those are bad examples, but they should illustrate the point. Over engineering is a real phenomenon.

Re:Technical Debt: Why it'll ruin your software by kazinator on Friday October 23, 2020 @01:46PM (#24871978)
The definition of "technical debt" is so pliable that we can easily regard all the parts of the system that are overengineered to be a form of technical debt.
Re:Technical Debt: Why it'll ruin your software by hmillison on Friday October 23, 2020 @01:55PM (#24872068)
Agree with this 100%. I've seen both in older codebases.

It's even more frustrating when you work with engineers who refuse to believe that any code they write could become technical debt in the future. These tend to be people who overcomplicate systems to anticipate future requirements

Re:Technical Debt: Why it'll ruin your software by Aeolun on Saturday October 24, 2020 @12:26AM (#24876932)
> These tend to be people who overcomplicate systems to anticipate future requirements

I don’t think I’ve ever anticipated a future requirement that did not ultimately turn out to be necessary.

Conversely, I’ve had a lot of people tell me something was not necessary only to find that, surprise, surprise, it was necessary after all.

Re:Technical Debt: Why it'll ruin your software by paiute on Friday October 23, 2020 @03:11PM (#24872772)
“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.” ― C. A. R. Hoare

This is how I approach software, but there are so many people out there who won't approve simple code, because it doesn't have enough configs, or classes, or whatever. It seems that the person who wants the most over-engineered code tends to get their way. Psychologically the absence of 'things' is always inferior and harder to argue for than having more 'things'.

Re:Technical Debt: Why it'll ruin your software by Gibbon1 on Saturday October 24, 2020 @05:30AM (#24878010)
The mode code you have the mode code you have to maintain.
Re:Technical Debt: Why it'll ruin your software by lalalandland on Friday October 23, 2020 @03:15PM (#24872805)
C.A.R. Hoare: 'There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.'
Re:Technical Debt: Why it'll ruin your software by asiachick on Friday October 23, 2020 @02:43PM (#24872504)
I'd call React and Redux both massively over engineered. There's a ridiculous amount of lines of code being executed to do the simplest of things. You do something like `a.b = c` and really `a` is a proxy so that setting `b` the setting gets tracked going through 100+ lines of code all of which has tracking structures that get setup and torn down between components. They claim this will make your life easier but when the facade breaks you end up having to design your entire app around the fact that `a.b = c` is not simple, it's complex and so you have to avoid it as much as possible my piling on more complexity.
Re:Technical Debt: Why it'll ruin your software by danenania on Friday October 23, 2020 @03:06PM (#24872724)
A simple react/redux app is definitely much harder to understand than a simple vanilla js app. But a large, complex react/redux app is far easier to understand than 99% of large, complex vanilla js apps.
Re:Technical Debt: Why it'll ruin your software by yoz-y on Friday October 23, 2020 @02:59PM (#24872670)
All GUI frameworks hide a lot of what’s happening in the insides. There is a scale to it but personally I think that reasoning about a “reactive” program is quite easier than untangling a ball of callbacks written in say, GtK.

Reactive frameworks indeed hide a lot of complexity but that does have positive impact on the user code. (IMO)

Re:Technical Debt: Why it'll ruin your software by barryhoodlum on Friday October 23, 2020 @07:33PM (#24875532)
Neither React or Redux use proxies or setters at all. Are you thinking of a different framework?
Re:Technical Debt: Why it'll ruin your software by asiachick on Saturday October 24, 2020 @04:06AM (#24877698)
some lib recommended to remove all the redux boilerplate of react, uses proxies. It's mostly irrelevant though, the point is there's too much code running to do what should be simple things.
Re:Technical Debt: Why it'll ruin your software by hinkley on Friday October 23, 2020 @02:42PM (#24872501)
I think I write code the way people describe Jazz. You have to listen to the notes he isn't playing.

I leave breadcrumbs and openings in my code all the time so that the next time I'm in there, I'm set up, invited even, to ask the question I left unanswered before, and maybe do something about it. I leave the option of adding a feature, instead of building a config framework to support it and then defining one implementation. Which will end up not being the correct API when (if) I write the third one.

Sometimes people beat me to it, and get excited because they had an idea they're now invested in. I usually just let them have it, don't point out that I put the idea in their head. It's so infrequent people get that invested in functionality, you have to encourage it. There's too many things I'd like to do and never enough time anyway. They've just nominated themselves as a candidate for maintaining that module when I get tired of looking at it, freeing up my time for things nobody else cares about until it's done.

It's like if you were designing a house with plans to expand it later. You'd be very careful about certain decisions, like whether you should put a bedroom in the logical spot for an addition, because now that bedroom also has to be a thoroughfare. For 20% extra effort, you've extended the calcification threshold for that house/code by 80%.

Re:Technical Debt: Why it'll ruin your software by dudul on Friday October 23, 2020 @01:56PM (#24872075)
Totally agree. As long as the system is easy to change, I don't care if it's "technical debt". I'll even go further, I don't care if it doesn't give us "flexibility" and "modularity". I don't care if it's not "generic". There is nothing that pisses me off more than people designing systems for future requirements that are not even being discussed by product leads.
Re:Technical Debt: Why it'll ruin your software by pknopf on Friday October 23, 2020 @01:41PM (#24871923)
Also, under-engineering can sometimes mean writing more code, or take longer to implement.
Re:Technical Debt: Why it'll ruin your software by marcinzm on Friday October 23, 2020 @01:45PM (#24871970)
Which may be a great thing if it saves you time refactoring or modifying code down the line. The hardest thing about code is not writing it but reading it, and clever succinct code can be very hard to read.