But your overall point is very correct. Don't make an abstraction because of coincidence.
I'm not sure I agree with this. For me, the point of abstraction is divide the number of concepts between the layers you introduce, effectively to hide concepts from the layers where you don't want to have to care about them. Often times, abstractions adds the total number of concepts at play, but hides them beneath/above the layers.
But if that's going to happen, then in the early days, a little duplication was probably better.
instead of “duplication is cheaper than the wrong abstraction”, I would say “duplication is cheaper than confusing code littered with conditional logic”.
The truth is that the phrase "wrong abstraction" is (more or less) unquantifiable, which makes the original phrase, as employed, sort of like a koan. It addresses the very human tendency to see patterns in noise, and our ability to "transmit" such hallucinations to other humans via natural language and other means.
The closest I can get to - given my at-best-apprentice status as a formal programmer - is the quantitative test I developed for CCS (conditional content systems), where the abstraction lies in the SNS[1], and the de-duplication mechanism is applicability[2]. Since each applicability statement carries its own overhead, there's a limit on how much "abstraction" the model can take before it's using quantitatively more keystrokes than duplication.
The test goes like this: take the flat text procedures for ALL the configurations, and add it together. Now, take the conditionalized, applicability-laden procedure that unifies the procedure, and measure its file size. If the latter is LARGER than the former, then you're using the wrong SNS/applicability model for rolling up this content.
Thing is, this is inevitable if you throw enough dissimilar configurations at a CCS, because each configuration has its own overhead, and eventually that outpaces the content itself.
You can address this in a bunch of ways - like adding a containing pseudo-product that has all the configurations inside of it - but the actual real Product Management might not let you build on the applicability like that, because the Product itself isn't sold that way. Any other abstraction isn't available to you, because in the end this is natural language, which - unlike structured language - resists first order abstractions really well. This is one of those instances where, yes, the abstraction of the SNS/Applicability is worse - quantifiably - than duplication. All that complexity would be better handled via version control fork/branch relationships - far outside of the realm of natural language.
[1] standard numbering system, a sort of numeric designator of functional systems, the primary way that content is designated as semi-independent modules.
[2] conditional "chunks" that turn on and off depending on the applicability statement
> All that complexity would be better handled via version control fork/branch relationships
Probably not.
Re: fork/branch shenanigans, no, you're right, that's not an optimal way to handle variance . . in a normal programming language. In the context of natural language, it's not the same kettle of fish, because, well, lots of reasons, probably the most prominent being the "messy unidirectionality" of NL that's all mish mished with its extremely complex grammar vs constructed languages. Chopping up giant documents into tiny pieces a la CCS[1] systems has made this a stew of problems, but for some reason Leadership is fond of the idea. It's not unlikely that specialized on-prem LLMs are going to nuke the CCS concept from orbit in the next five years, except for those cases where the CCS is a contractual requirement for doing the work.
[1] component content systems
This seems to be the underlying assumption behind most uses of the "duplication is cheaper than the wrong abstraction" quote, but the assumption is simply incorrect. You should almost never try to expand abstractions in this manner. If you don't treat the abstractions relating to the thing you want to change in your codebase as "the" place where you need to make your change, and instead eagerly make new abstractions and throw old ones away as required, you won't really run into this problem.
In fact, this predominant mindset where creating abstractions is strongly discouraged leads to the very problem that mindset is based on, as it will simply encourage junior developers and the like to modify the existing abstraction, creating the aforementioned kind of mess where abstractions become complicated through repeated modification, instead of creating new abstractions when appropriate, because creating abstractions has a stigma attached to it.
Additionally, if someone has made a "wrong" abstraction based on something silly like two pieces of code simply being similar in terms of their structure and those use cases start to drift apart, you should feel eager to simply split apart the abstraction, be it into bare implementations or two new abstractions, or any other combination. Abstractions are cheap as long as you don't give them special significance.
you will need to replace that abstraction with duplication.
Which is the right thing to do because that duplication is cheaper than maintaining the wrong abstraction.
I think this post makes the mistake of thinking that the only way in which duplication comes up is that it is discovered in the codebase, and we have the choice of abstracting it away or keeping it.
On the contrary, duplication can - and should - be consciously introduced to fix bad abstractions when we find them in the codebase.
Hard disagree. When the formerly common parts of an abstraction evolve to no longer be common, then that duplication no longer exists. There now exists two abstractions, one for each of the diverging needs. There may be some leftover commonality that can be abstracted out, but it's no longer the original abstraction.
You're saying we should look for duplications, abstract them, and then every time a change needs to be made to the abstraction to suit only one of the use cases, refactor the codebase to de-abstract and re-duplicate, undoing the work we did in the name of DRY in the first place.
That is a lot more work and a lot more confusion and a lot more headache for maintainers and reviewers than copy-pasting the thing the first time, having realized that the duplication was incidental, not structural.
Let's take this line of reasoning to its extreme:
I notice that there's a section of my code that's repeated twice where we add one to a value, so I abstract it into a function called add1(x:int). Some time later, at places where add1 is used we sometimes need to actually add a value other than one, so we need to make a decision: do we refactor everything and re-duplicate, or do we stick the DRY principle and make our abstraction more accomodating? The path of least resistance is to stick to DRY because it's a smaller and more comprehensible commit, so we add an optional arg, add1(x: int, operand?: int). Some time later one of the callers to this function needs to pass a vector instead of a single value, so we need our add1 function to have polymorphism and conditional logic in it now, and potentially more arguments. Sooner or later we have a frankenfunction that's hundreds of lines long and branches a bazillion ways and might as well be a turing machine in itself.
Dogmatic adherence to DRY leads to madness.
That's the exact opposite of what I'm advocating for, but perhaps I didn't express myself well.
> Sooner or later we have a frankenfunction that's hundreds of lines long and branches a bazillion ways and might as well be a turing machine in itself.
Yeah, that's not a good abstraction, and not at all what I meant.
Instead, any given task (which already is an abstraction) should exist in only one place. That is DRY, I would paraphrase it to mean any given abstraction should be done in one place (and combine with SRP to say further that one place should only do that one abstraction)
If one place can be updated independently of another, it argues it is not the same task to begin with. DRY'ing that code is a misnomer IMHO, instead that code is being put behind a procedure and is being made generic (and not necessarily more abstract. Abstracting hides details, putting a block of code behind a procedure with full parameterization is not hiding details, it's just a procedure [and let us hark back to the days of procedural programming and ways that can become mess])
DRY and SRP (single responsibility principle, AKA the DnD principle) need to be considered together.
Applying DRY primarily to structural duplication never occurred to me, and only this discussion brought this way of thinking to my attention. It's always been about semantic duplication to me. Often, but not necessarily, semantically duplicate code has structurally duplicate lines of code.
But now I think I understand why some junior programmers I consulted for denied that their system was rampant with the duplication I was seeing. To them, the code was different because it looked different; that it all was doing more or less the same thing, but with different inputs, seemed lost on them. I have a vague recollection of one of them saying something about "parameterizing" the code, and then dismissing it.
I'm going to have to dig through my notes from that gig to see if I can better clarify to myself the implications of the focus on only structural duplication and why it might lead a programmer to overlook very obvious opportunities for removing semantic duplication, or not understand how to fix it even if they do see it.
What piece of knowledge does your add1 function represent? I don't think you're strawman is actually DRY (which is a problem - damp strawmen can mildew).
I do agree that there are sometimes tradeoffs that can make a less DRY approach a better one, but not all deduplication is "DRY".
Every element of data, core capability, or logical object should have a single authoritative source. Having multiple db connections or ui controllers would, in fact, be a horrible nightmare, obviously.
But not all code is knowledge, some of it's just boring work that does something to some local stuff and it doesn't necessarily need to be sharable with other parts of the code base, just because the logic looks kinda similar.
The common (mis?)understanding of DRY as primarily syntactic both overshoots and undershoots. As many here have discussed, it can lead to combining things you shouldn't (I jokingly call this "Huffman coding"), but it can also fail to recommend combining knowledge when it is represented differently. If I'm saying "there is a button here" in my HTML and in my CSS and in my JS, that's three places for that piece of knowledge even if those three places don't look anything alike. Changing the CSS to "here is what my buttons look like" and the JS to "here is how my buttons behave" would be DRYer.
This is basically introducing the duplication back.
Whether you keep the duplicated code or refactor it in a different way is another question, what matters for the "duplication is cheaper than wrong abstraction" to be true is just the fact that by introducing abstraction early you wasted time refactoring one way and back. Refactoring isn't free. So in fact leaving the duplication there would have been cheaper - Q.E.D.
It doesn't mean you should never risk it, but it does mean you should think hard before you do it.
But somebody on your team /will/.
> you should feel eager to simply split apart the abstraction
Sure, but it's going to be a lot more work at this point than if we had avoided the mess in the first place.
That's not to say abstractions never change -- they do. But they change because your understanding of the sub-problem they're solving has changed, not because their implementations or consumers have changed.
I could see this also happening in situations where such problems arise at component interface boundaries, where the pressure not to change comes not as an organizational policy but rather inability to influence external components (eg. the OS offers a poor abstraction -- the user-space program developer wouldn't be changing how OS works to rectify things). But, this is again sort of an administrative problem, because, ideally, the user-space program developer should be able to convince the OS developer to change the interface if it's found to be not a good abstraction...
But, yes, I can see how in practice that'd be a very difficult thing to do.
yeah - if an application company runs up against a poorly designed OS interface - they usually aren't aware, or just back away slowly. they don't have the scope or mandate to pursue it.
that kind of behavior often extends to library dependencies, and even internal interfaces that the company should ostensibly own.
its not so much an administrative problem as a desire to limit spending and time on software and do a kind of agile mvp glue job over an unbounded number of external dependencies. that often leads to an unmaintainable hairball. but if the alternative is to hire a bunch of really experienced developers and let them 'do the right thing' for 5 years...
even if you do, there is a still a good chance that the output isn't gonna be that great. software people lost a lot of credibility with that model. we've probably swung too far in the other direction
Then you split that abstraction again. It's very cheap and very quick.
Many people talk about the issue like it was an absolute in the code, but that's wrong approach. If you end up writing 4 functions that are the same, by all means, merge it into one.
If then you need to add a parameter only this code path uses and rest doesn't care about, by all means split it back. Moving blocks of code around is cheap.
However, once abstracted, the code may suddenly be used by a number of different teams. You will need to get this work on their roadmap, increasing the friction to get this done. In many companies, this will also end up in endless discussions about the new approach.
If you have a mixin (or other means of composition) that you use in several places and one diverges, it's easy to remove it. If you use inheritance, it's going to be more painful.
A language that offers OOP via prototypes instead of classes like JS can (sometimes) give you the best of both worlds, but it will confuse a lot of devs who aren't familiar with that kind of OO design.
At a technical level an abstraction will have more call sites to worry about in different contexts, the more wrong the initial abstraction the harder it will be to change.
The social level is maybe even more problematic. Abstractions seem more important than calling code and will experience more friction in code review. This change friction can also increase with the "wrongness" of the initial abstraction. The starting point makes less sense so a reviewer needs to work more to understand the context. If the abstraction is gnarly enough then it's possible that the reason for the abstraction is almost obscured. Even someone who knows _how_ it works might have lost the forest through the trees and push back on changes that simplify it or improve it if the change is a sufficiently large departure from the initial state. In this case you can often see small incremental changes get added easier but this just makes the shared code a bit gnarlier for next time.
Usually what happens is the 'clean' code ideal comes first, and then the implementation is squeezed into it. This then informs the organisation (or architecture) of the rest of the codebase and your software design has become a matter of putting pegs into the right-shaped holes.
I have never found that kind of highly abstracted code easier to work with than some simple procedural alternative that is easy to delete and easy to refactor, so long as effort was put into writing it well.
Of course, the patterns have a purpose and do help when used nicely - a lot of code you write will fall into some of those patterns even without you explicitly mentioning it. It's just...doing it for the sake of it is a problem.
The pile of abstractions stacks to the moon but when you chase down what's actually happening a 60 file repo ends up holding like 12 lines of actual programming.
As I recently called it, infrastructure and systems lose agility as they gain dependency and move down the stack.
If you have like 1 customer and they have good retries, honestly: fuck everything. Deploy master, in fact, deploy every keystroke to prod. It'll be fine.
At the same time, about 30k - 40k FTEs of our B2B customers depend on one of my Postgres instances during business hours and about twice of that during different holiday seasons. Honestly? Nothing touches the system-level settings of these database systems unless we have pondered a change for 2 weeks. And even then we will schedule an approved change over 4 weeks across applicable postgres clusters. The carnage a bad change at this level can cause is ridiculous enough to not be.
In my experience, the right way to handle this sort of situation is to create a functional mini-DSL for the process that handles all the implementation details, then create a "default" process which serves as a template. If a process needs slightly different logic, just copy the template, update the DSL to support any new logic, and update the template with the new DSL statements. This approach lets you give semantic meaning to implementation details, and you can see where all the different custom logic is at a glace by looking at all the template copies. As long as the template is only calling out to DSL actions with no internal logic of its own and process flow is correctly encapsulated in the DSL, you should never need to update templates to change behavior, only update the DSL.
Exactly. Is there a formal term for this?
Instead of one gigantic function with 50 parameters, you have 100 "template" functions, that all make use 60 different "helper" functions (what you're calling the DSL).
Instead of castles-of-logic abstraction, it's nuts-and-bolts or grass-roots abstractions. I've never come across a name for this development style.
But it generally works extremely well when building processes for tens/hundreds of data formats or customers or what have you.
Libraries designed like this are sometimes called "combinator libraries".
Always a good idea to expand an abbreviation the first time it's used.
Though it's less true today and in languages that are not C or Fortran. Even something like C++ or Java has the template method pattern, which gets you 80% of the way there. Dynamic languages like Python or Ruby tend to have pretty reasonable facilities for building DSLs, as do more modern languages like Scala and Rust.
So in your particular case, it could have been possible to abstract away the code at that point in time and once they diverge, remove the abstraction and duplicate, then adjust one of the duplicates (which no longer is a proper duplicate really).
But, might be more work than it's worth. YMMV.
> So in your particular case, it could have been possible to abstract away the code at that point in time and once they diverge, remove the abstraction and duplicate, then adjust one of the duplicates (which no longer is a proper duplicate really).
This sounds nice in theory, but the reality is that the effort required to make these two kinds of changes is not symmetric. It's about 10 times easier to get a PR approved and merged that combines similar looking code into a function than vise versa. If you any suspicion at all that an abstraction you're making may need to be removed and duplicated in the future, you're better of just never abstracting in the first place.
It sucks pushing a change which unwinds an abstraction like that through code review. It's usually a lot easier to just never abstract it in the first place.
Sometimes duplication is cheaper than the wrong abstraction.
And
Sometimes it's better to abstract away a duplication rather than let it lie.
And that's the mark of becoming a master at the craft. Being able to recognize all of these various slight permutations of state and what to do about them.
Repeat yourself to avoid creating dependencies, but don’t repeat yourself to manage them.
I'm not against shared test fixtures or some utility functions, but IMHO, it's better to have some duplication but clearer tests.
There will probably be some duplication across tests, but if the utility functions are idempotent/composable, they're usually pretty easy to read/understand and equally mechanical to write/update.
I would add that you should duplicate the common, cross-cutting setup (eg. faked/mocked dependencies that don't matter), but make the test conditions themselves explicit.
You get a feel for the correct granularity the more tests you write within the codebase. If you try to be too clever in saving boilerplate, you'll cause pain for future modifications and maintainers. Sometimes fixing "clever" tests takes longer than the code change itself.
I think it depends on the context. For example, I typically agree, but when I was writing authz tests [0], I ended up writing a DSL so that 1) I'd more more inclined to write the thousands and thousands of tests, and 2) I'd be able to focus on the actual authz assertion and not on verbose setup.
I couldn't imagine writing those policy tests without that abstraction. I would have lost my mind with all of the repetition, and would have almost assuredly made mistakes.
[0]: https://github.com/keygen-sh/keygen-api/blob/master/spec/pol...
It's not the prettiest when you actually look beneath the covers [0], but it does what I wanted -- providing an easy way to write exhaustive authz tests. Without the DSL, I probably wouldn't have written the tests. The PR for said migration was massive [1], and was a prerequisite to going open source [2].
[0]: https://github.com/keygen-sh/keygen-api/blob/master/spec/sup...
In that case, my takeaway would be that it ain't the abstraction itself that's wrong, but the unwillingness to get rid of it (or decompose it) when it no longer serves its purpose.
The answer isn't to not abstract, the answer is to tear it out when it turns wrong. That was actually the original point of the popular article that streamlined this view - that we shouldn't be afraid of tearing them out, not that we shouldn't make them in the first place. Most people just read headlines though.
While the functions are distinct the call sites are self-documenting. You know which calls are for which purpose because the names are different. After combining them to deduplicate the code, you've lost that information, and to disentangle the abstraction now requires you to infer and reintroduce that lost information.
It's not that it can't be done, but there is real friction that doesn't just exist in people's heads.
I wrote a short post in roughly this idea space last year: https://t-ravis.com/post/doc/what_functions_and_why_function...
It feels like the same thread you're describing, but I guess it's pulling on the other end of it. It's thinking about how to name things in a way that makes it easier to see that the implementations might diverge later, and simplify actually doing so (by preserving more of this intentional context).
In contrast, duplicated code may be annoying, but it is usually simple to understand and maintain by anyone with knowledge of the project and the programming language.
Leaky abstractions are way way way worse than no abstraction at all.
This is bullshit. I think what you mean is that even a good abstraction can be used inappropriately.
You shouldn’t compound that with the fact that sometimes people create abstractions that are just bad.
I think how much you hate that may depend on your language and the program. Some big enterprise Java monolith is a garbage dump of thousands of small files. So who cares. In C without name spaces and the need for headers you care more.
The other one I hear a lot is "it's not realistic to reach 100% test coverage / type safety" when submitted code with `any` all over it and zero tests.
I disagree with the severity of this, and would posit that there are duplications that can't be "fixed" by an abstraction.
There are many instances I've encountered where two pieces of code coincided to look similar at a certain point in time. As the codebase evolved, so did the two pieces of code, their usage and their dependencies, until the similarity was almost gone. An early abstraction that would've grouped those coincidentally similar pieces of code would then have to stretch to cover both evolutions.
A "wrong abstraction" in that case isn't an ill-fitting abstraction where a better one was available, it's any (even the best possible) abstraction in a situation that has no fitting generalization, at all.