News for hackers. Stuff that matters.
Re:Software Architecture Is Overrated, Clear and Simple Desi... by JMTQp8lwXL on Wednesday September 18, 2019 @02:55AM (#21002941)
Over my career, I've worked with engineers that like to over-engineer and under-engineer.

The over-engineered code looked like russian dolls: had many layers to it, and some of the abstractions offered no value. That can make onboarding to such code unnecessarily complex.

On the other hand, under-engineered code made very little of use of even simple data structures or algorithms. I like to call it "chicken scratch" code. I find it tends to be brittle, and it fails the longevity test: you end up frequently having to touch everything, and be aware of the entire state of the system at once, due to a lack of functional style. There are few enforced boundaries between subsystems in this type of code.

Like most things, moderation is key. I only introduce abstractions when there is a meaningful value-add to doing so. This is somewhat subjective, but there is a right level of application of design patterns. Not too much, nor too little.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by stinos on Wednesday September 18, 2019 @03:46AM (#21003152)
In my own experience, it's really a must to go through both over- and under-engineering phase yourself: only then I found I got a really clear view on what's good and bad in approach, allowing me to default to somewhere in between (with outliers when useful) and in any case with focus on simplicity (but again, if needed for e.g. performance that can be violated). All of this is the basic 'learn from your mistakes' principle in the end - and it works.
Re:Software Architecture Is Overrated, Clear and Simple Desi... by fnord123 on Wednesday September 18, 2019 @05:01AM (#21003471)
I don't even think it's a phase.

From what I see in my own experience, overengineering comes from not having a deadline to deliver an actual working piece of software. Unclear requirements. Fear that the code must be perfect against change without knowing what the changes might be.

Underengineering comes from confidence that the issues are too small to worry about. Or the time constraints mean I can't try to find good primitives to make something robust. And if I have to return to it because it's so important, I can re-engineer it then.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by tluyben2 on Wednesday September 18, 2019 @05:33AM (#21003596)
I have been writing software professionally for almost 30 years now and yes, this is exactly what I see too. I find the underengineering far more productive though; because of time constraints I hack something together which works but is brittle (decades experience do help fight the brittleness even in a hurry though) and I do revisit to fix and refactor if the software survives.
Re:Software Architecture Is Overrated, Clear and Simple Desi... by fnord123 on Thursday September 19, 2019 @03:22AM (#21014063)
Underengineering is far more productive but not always possible when you're defining the overarching architecture of a multi team project.

Then you have to rely on experience to do things like definelibg interfaces that can be extended later on instead of overspecifying types. It's hard to balance it just right.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by stinos on Wednesday September 18, 2019 @06:31AM (#21003860)
overengineering comes from not having a deadline

This really resonates: at the height of the phase I spent about 2 years once, half-time developing something new for which there was essentally no deadline. I went totally berserk doing insane stuff like implementing design patterns using templated 'reusable' classes in C++ so whenever something even remotely looked like a pattern I had a class for, I would use that class. Which then needed small fixes of course. And extra logging. Pluggabble. Etc, you get the point :) In the end everything was dropped because I found a much better, much simpler, existing way to do everything though.

But didn't happen to me again afterwards, or at least not to that extent: it was really like having had a severe electric shock; once, but never again, before even coming close to any hot wire my brain is already subconsciously screaming to get away from the thing a.s.a.p.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by delinka on Wednesday September 18, 2019 @11:41AM (#21006532)
Overengineering: I just left an enterprise-y company where the lead software engineer "over-engineered" for the purpose of job security. His philosophy was that if it's layered and convoluted, it's harder to replace him. Oh, we had deadlines. It had to work. Part of "working" is "passing tests" so he also wrote lots of tests that executed the written code but actually tested the mocking framework.

I hated dealing with any of his code.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by psychoslave on Wednesday September 18, 2019 @08:04AM (#21004399)
My experience is that you rarely have the time to make things right, you take the time. And you do so, because you estimate that despite doing so will put you off the deadline, you expect it to be a good bet on the long run.

Not doing so, time required for maintenance will increase, and competing with new feature development time. Of course these new features have deadline already to short even before you add this burden. Moreover, maintaining everything working is the top priority "right now", but making the new development reaching production is prevent the solution to fall into market irrelevance.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by schwurb on Wednesday September 18, 2019 @05:57AM (#21003697)
I never went through the over-engineer phase personally. I am a very very lazy person, which leads to me to correct balance most of times: Not enough abstraction, I suffer because of too much change to the code base, too much abstraction, and I feel pissed about my risk management. Since I suffer greatly when I suffer, I avoid suffering in my own code as much as possible. What also helps is that I plan most of my algorithms and program designs on paper, actually touching a computer is the last step in that process.
Re:Software Architecture Is Overrated, Clear and Simple Desi... by stinos on Wednesday September 18, 2019 @06:18AM (#21003806)
too much abstraction, and I feel pissed about my risk management

Interesting, so it's not a phase for you, but for the rest it sounds exactly like my point, i.e. learning that over-engineering (too much abstraction) is a mistake and as such trying to avoid it.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by schwurb on Wednesday September 18, 2019 @08:41AM (#21004704)
Yes, we are on the same page there. I think due to depression during my formative years, I have a mix of slight apathy towards expectations and a slightly off motivation system, which leads me to be very lazy - really only doing what is necessary to get a mantainable product and not so much caring about what other people want. Working in two software companies during studies was okay, but I really hope to sell my own software one day!

Glad to see you making progress!

Re:Software Architecture Is Overrated, Clear and Simple Desi... by tarsinge on Wednesday September 18, 2019 @03:02AM (#21002982)
I think you can under engineer and still use a pragmatic functional style.

Also yes when it’s under engineered you usually have to touch multiple parts to make a change, but in the end it usually take less time (testing for bugs included since the code is simpler).

Edit: I don’t say under engineering is perfect, but compared to an over engineered code base I prefer the first for maintainability.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by JMTQp8lwXL on Wednesday September 18, 2019 @03:08AM (#21003010)
In theory, I agree with you, but in practice, I rarely see under-engineered code that is functional in style. My conjecture is that people under-engineering aren't even aware of the concepts of imperative vs. functional style.

The more we can think of code as existing in a sandbox that knows nothing about the surrounding system it's in, the better. Because when you need to change that code, you don't need to juggle comprehension of the rest of the system as you edit that code. It makes reasoning about the code much simpler. This is why I agree -- code can be pragmatic and functional.

But if you tend to practice having code existing in little sandboxes, you are essentially practicing design. This might be semantics, but now you have sub-systems with well-defined borders that interact with other sub-systems in proscribed, expected ways. I would consider that, to some degree, the application of design patterns and architectural concepts -- even if it's only the inevitable consequence of writing pragmatic, functional code.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by bayindirh on Wednesday September 18, 2019 @06:57AM (#21003966)
> I think you can under engineer and still use a pragmatic functional style.

I actually believe this, and tried in some occasions for my personal projects.

When you develop the code further, you need to touch everywhere. It starts very neat and being extremely tidy at first, but when you need to add some significant features, the design breaks down pretty badly.

Then, to save the architecture, some significant refactoring runs are required, and boy, these refactorizations are expensive both time and design-wise.

If your project will not add new features over time and relatively compact, under-engineering can deliver pretty nice results. Otherwise, designing with some headroom and flexibility goes a long way.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by jamesb93 on Wednesday September 18, 2019 @07:05PM (#21011429)
You should clarify that your experience us pretty much exclusive to web development which is bot indicative of a lot of work that goes on in software creation and computer science research. Just a weird nit pick of mine that most people here seem to be web devs who assume everyone else is - it creates a weird environment for discussion.
Re:Software Architecture Is Overrated, Clear and Simple Desi... by EGreg on Wednesday September 18, 2019 @10:56AM (#21006064)
In general I have found — over 20 years of experience architecting software - the following:

1. Platforms and reusable frameworks should be architected as well as possible. Apps can be whatever.

2. A developer who writes clean code and documents it is far better than a “10x” developer, unless you have budget for only one developer.

3. Functions should have extensibility, put the required parameters as parameters and always include an “options” at the end. Each function can have defaults that you can extend, which means you need a deep-extend method:

https://qbix.com/platform/guide/javascript#functions

4. When in doubt whether to do convention A or B, take a bit more time to do C which can handle both, and add the convention in a config. You never know when someone may need something else!

5. Similar to 4, if you can have an extra indirection, add it. So you can let others hook into “before” and “after” hooks at any point. Use events instead of functions.

6. In fact try to have event driven architecture rather than futzing around with mutating data. The easiest way to sync is to maintain a linear total order for events.

7. Think about how lookups will proceed and partition everything by those keys. Sometimes you need to have duplicate tables and keep a sync from a “primary” table to a “secodary” one in the app layer. Doing this allows you to do sharding or even go serverless peer-to-peer later!

8. Security: more checks are better than less. Pile on private keys, bearer tokens (api or cookie), and so on. Use a device keychain:

https://qbix.com/blog/2018/01/17/modern-security-practices-f...

Re:Software Architecture Is Overrated, Clear and Simple Desi... by igor47 on Wednesday September 18, 2019 @11:50AM (#21006619)
This list is full of hilariously terrible advice. Was that intentional? On the internet, nobody know if you're a dog or being sarcastic, and I dread stumbling across a code base where someone took some of this stuff seriously.

Examples: > if you can have an extra indirection, add it.

> Use events instead of functions.

> You never know when someone may need something else!

Re:Software Architecture Is Overrated, Clear and Simple Desi... by EGreg on Wednesday September 18, 2019 @12:02PM (#21006759)
Please be specific about the issues and let’s discuss. I am serious - this list is optimized for maintainability of code. Developer time is more valuable than processor cycles, in most cases. Unless you are the kind of person who would argue that C++ introducing object orientation and virtual methods made everything slower and that extra indirection by default is hilariously bad architecture?
Re:Software Architecture Is Overrated, Clear and Simple Desi... by igor47 on Wednesday September 18, 2019 @12:26PM (#21006983)
> 3. Functions should have extensibility, put the required parameters as parameters and always include an “options” at the end. Each function can have defaults that you can extend, which means you need a deep-extend method:

counter-point: why not write functions that take the arguments they need? if they need more arguments later, why not add them later?

> 4. When in doubt whether to do convention A or B, take a bit more time to do C which can handle both, and add the convention in a config. You never know when someone may need something else!

counter-point: why try to anticipate future needs? maybe it won't happen. maybe it'll be something you totally don't expect. you should write code that's easy to understand and change, so it can react to future requirements. you should not write code that's bloated with unnecessary features, because it slows down your future ability to iterate.

> 5. Similar to 4, if you can have an extra indirection, add it. So you can let others hook into “before” and “after” hooks at any point. Use events instead of functions.

have you ever tried to figure out how a piece of functionality is implemented, only to go chasing it across 10 files because of indirection? have you ever had a piece of code that runs, but you don't know why it's running because you can't tell which things are firing what events? have you ever had to ask the question, "what happens when i execute this function", but been unable to answer because of hooks/observers that are not directly connected to that function?

my counter-point: use the barest minimum of functionality you actually need. stick with functions and maintain linear control flow with minimal side effects.

> 6. In fact try to have event driven architecture rather than futzing around with mutating data. The easiest way to sync is to maintain a linear total order for events.

if you have to mutate data, you should mutate data? not sure how events save you.

> 7. Think about how lookups will proceed and partition everything by those keys. Sometimes you need to have duplicate tables and keep a sync from a “primary” table to a “secodary” one in the app layer. Doing this allows you to do sharding or even go serverless peer-to-peer later!

wow! serverless peer-to-peer! i can't wait to re-write my app, i'm sure my customers will love the new architecture! /s -- on a serious note, while distributed sync is sometimes required (for instance, state kept in the JS front-end app that is a copy of the DB-preserved state on the backend), keeping them in sync can be a nightmare, and you should avoid this at all costs if possible.

> 8. Security: more checks are better than less. Pile on private keys, bearer tokens (api or cookie), and so on. Use a device keychain

in my experience, 80% of all bugs are security bugs (that number may be as low as 20% if you use the OPs advice to increase your bug count). while security is important, you should avoid "[piling] it on", and instead be thoughtful about it. separate authentication (accomplished with keys/tokens/etc...) from authorization (accomplished through checks of the identity at points of functionality).

Re:Software Architecture Is Overrated, Clear and Simple Desi... by EGreg on Wednesday September 18, 2019 @01:42PM (#21007892)
3. Because it explicitly signifies a place to put those later arguments, in a way that is MAINTAINABLE.

First of all, the last parameter should have a default value of {} — that is, no options, but can still be deferenced in code. So not passing options is ok.

Secondly, if you don’t do this, future developers will keep adding parameters in an ad-hoc manner until you get stuff like:

context.copyImage(src, dest, sx, sy, sw, sh, dx, dy, dw, dh, filter, rotation, matrix, ...)

and your calls will keep looking like:

context.copyImage(a, b, 0,0,30,30, 20,20,40,40, null, 5, null, “foo”)

Not only will it be harder to read for anyone looking at the calls, but also the future function signatures will have parameters in the chronological order they were added — which is often totally unrelated to the order they should be in, but you can only add them at the end.

Since the function should be backward compatible, all new parameters are by definition OPTIONAL and therefore can be added to a hash or object called “options”.

And YES I stand behind this. Years ago I actually recommended this to the PHP language mailing list:

https://grokbase.com/t/php/php-internals/1042mr8yrn/named-pa...

In other words, I wanted the function call syntax in PHP look like the array syntax:

func($a, $b, $c => 3, $d => “foo”);

Simple, and elegantly enforces the above convention while looking “quintessentially” PHP!

4. When you are building a re-usable framework, it’s silly NOT to anticipate future needs. The whole point of a framework is future needs.

Now, you say you should weite code rhat is easy to undeerstand and change. If you hardcofe vales, that’s easy to change - just put a variable there. But if you hardcofe a concention in 100 places, fhat’s not ao easy to change. You can’t just grep for a hardcofed constant.

But ot gets worse rhan that. Other code will come to rely on this “invariant”, which may change later. Again, the wholw goal of my recommendations is to future-proof your code so that future developers will write code rhat grows up around it and can go in any direction, and can play nice with each other.

5. Suppose you are lacking middleware between A and C. So now you want to mock an input from A. Too bad, you can’t. Ok, what if you want to modify something that goes on between A and C? You have to rewrite A or C.

Let me give a real example. Suppose I said that people may have one more articles they write. So I implement a User table and an Article table, which has authorUserId field on it. Simple, right?

Except it’s too simple - one article can at most one author. If instead I had thought ahead and made three tables: Author, Article, and Authorship as a join table with articleId, authorId, then I could have 1 to N mappings in both directions and far more flexibility.

Now you may say — why think ahead? Maybe in the future articles can have more authors and THEN we will refactor the code! Except at that point you’ll have tons of plugins and apps, some beyond your control to change, relying on the details of your implementation.

Of course, you should also use another principle I didn’t mention (because it’s very obvious and popular) namely to write abstract interfaces that don’t leak implementation details, and keep these interfaces as small as possible, so that you can reason about large systems through these “bottlenecks”. But I have found that, on the back-end, it’s just a bit of extra work to add an extra indirection, whether you use it now or not. Instead of saying “we will NEVER do it the other way because it makes no sense”, if it costs you so little, why not allow for it, in case later someone will want it? The interfaces are often a leaky enough abstraction for this to matter.

For example you’d have article.getAuthor() if you didn’t make that extra table join indirection. And now what will article.getAuthor() return when articles can have many authors? It would return a random author, for backward compatibility. With my approach, you would prevent the “older” apps from using dumber interfaces. It’s just a bit extra work for everyone, for huge wins later. And that’s the point.

7. Event streams can be abstracted into pipelines and middleware. You can do undo/redo, store histories, have Merkle Trees and more. Compare SVN and Git :)

8. Sync becomes much easier when everything has a history of states. Look at git. You can just use it. Look at scuttlebutt, blockchains, or other types of merkle trees. Everything becomes super simple to reason about.

This forum could be refactored to be distributed. Everyone owns their own node of the tree and the relationship to its children (replies), and everyone else just replicates them scuttlebutt-style. Expanding a tree is fairly simple, and each branch has a merkle hash. There is no central server.

And the best part - you could start with a centralized app and gradually move to a decentralized, end-to-end encrypted model, if you only had the foresight to make sure that your tables has primary keys corresponding to how people look up some data (node, etc.)

The only thing you’d need to have consensus about is the ordering of replies to a node. And that consensus can be among the repliers or simply dictated by the parent node’s owner.

8. Having a non-extractable private/public key be used to sign requests is better than JUST having a bearer token (cookie). If someone commandeers the cookie via, say, session fixation, they still won’t have your private key. But they need the session id (bearer token) to look things up on the server. This is “piling on”.

Then, on top of this, you can have a blockchain of keys for devices, stored across sites, so you can revoke a device when it’s compromised, or authorize some new device with N previous devices.

You can have the same exact mechanism manage users in a chatroom or other merkle tree structure. This is what keybase does.

You can encrypt data on the server, with people’s public key, and they have to decrypt it.

You have to make sure that the initial signup requires some sort of token to prevent sybil attacks.

In short — once again the approach is to “layer on more security mechanisms”, they should all work independently.

You don’t just rely on HTTPS for example, because a server or CA cert can be compromised. You hash passwords on the client with salts before sending, regardless. Once again this is called “defense in depth”.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by dragonwriter on Wednesday September 18, 2019 @03:55PM (#21009362)
> Secondly, if you don’t do this, future developers will keep adding parameters in an ad-hoc manner

You seem to presume a situation where you have absolute control over the initial signature of functions added to the codebase but no ability to constrain future changes.

1. All parameters that for which a default makes sense should be optional and have a sensible default.

2. (In languages where this is an option) All parameters with a default must be keyword-only.

3. All new parameters to an existing function (from a stable release) must have a sensible default.

> When you are building a re-usable framework, it’s silly NOT to anticipate future needs. The whole point of a framework is future needs.

The point of a framework is to avoid solving the same problem multiple times in each new project. If you haven't needed to solve it twice in two separate projects, be skeptical that you need it in a framework. If you haven't needed it once, don't even consider it. Code for problems you don't have is pure waste.

Much of the advice you offer is going to produce waste because YAGNI; each time they come up the cost may be small, but in aggregate it's going to be a lot of extra zero-value code being written and maintained, bloating development and maintenance costs and timelines. Occasionally, you'll be benefit a little down the line from hitting a problem you correctly anticipated, but often you’ll suffer from having not having anticipated the real constraints of the problem when dealing with it when it wasn't a real need, meaning you’ll have to throw away your just-in-case code anyway, and all the time you'll be dealing with extra complexity dealing with problems you haven't had any real need to address but only imagined might come up in the future.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by darkarmani on Wednesday September 18, 2019 @02:02PM (#21008137)
> 3. Functions should have extensibility, put the required parameters as parameters and always include an “options” at the end. Each function can have defaults that you can extend, which means you need a deep-extend method:

No! Absolutely not! There is a time and a place for this, but it's nearly impossible to reason about the interface if any data can be passed in.

Re:Software Architecture Is Overrated, Clear and Simple Desi... by EGreg on Wednesday September 18, 2019 @03:34PM (#21009111)
I think you misunderstand. It’s not that “any” data can be passed in. The options object is documented in every version. It’s just a place that lets future versions add named parameterd
Re:Software Architecture Is Overrated, Clear and Simple Desi... by Madeindjs on Thursday September 19, 2019 @05:13PM (#21020822)
IMO If a method take many parameters there is a chance that this methods have too many responsibility and can be cut into two methods.
Re:Software Architecture Is Overrated, Clear and Simple Desi... by olalonde on Wednesday September 18, 2019 @01:20PM (#21007635)
Not sure you are aware but it seems you are describing "event sourcing" in point 6. [0]

[0] https://martinfowler.com/eaaDev/EventSourcing.html

Re:Software Architecture Is Overrated, Clear and Simple Desi... by xcubic on Wednesday September 18, 2019 @07:29PM (#21011621)
From someone doing webdev but without studies, is there a small list of simple data structures or algorithms that you would recomend?