That is a good rule of thumb, and I often follow it too. But it does take some discernment to recognize cases where something would benefit from an abstraction or some common code, even if it is only used twice.
I used to work for a company that imported airspace data from the FAA (the US Federal Aviation Administration) and other sources. The FAA has two main kinds of airspace: Class Airspace and Special Use Airspace.
The data files that describe these are rather complex, but about 90% of the format is common between the two. In particular, the geographical data is the same, and that's what takes the most code to process.
I noticed that each of these importers was about 3000 lines of C++ code and close to 1000 lines of protobuf (protocol buffer) definitions. As you may guess, about 90% of the code and protobufs were the same between the two.
It seemed clear that one was written first, and then copied and pasted and edited here and there to make the second. So when a bug had to be fixed, it had to be fixed both places.
There wasn't any good path toward refactoring this code to reduce the duplication. Most of the C++ code referenced the protobufs directly, and even if most of the data in one had the same names as in the other, you couldn't just interchange or combine them.
When I asked the author about this code duplication, they cited the same principle of "copy for two, refactor for three" that you and I approve of.
But this was a case where it was spectacularly misapplied.
What I like to do instead is break the shared code paths into a palette of smaller subfunctions/subcomponents and then have each use case have its own high level code path that picks and chooses from these subfunctions: One does ABCDE, another does ACDEX. It makes it supremely easy to reason about what each of them actually do, because they read almost like a recipe. It becomes a sequence of high level steps, some of which are used by several use cases, while others are unique. I've found this way of generalizing is almost "cost free" because it doesn't really couple things at a high level, and it's the kind of readability refactor that you'd often want to do anyway even if the code wasn't being shared.
Isn't this just the Command pattern? - https://en.wikipedia.org/wiki/Command_pattern
The most important (though not only) issue with frameworks is that you typically can't compose/mix more than one together - every framework is "exclusive" and takes control of the code flow. Whereas "components" can usually be easily mixed with each other, and leave control of the code flow to the programmer.
Couldn't disagree more tbh. Some of the worst code I've ever had to work with has been over abstracted "recipe" code where I'm trying to descern complex processes based off two word descriptions of them in function names.
Doing this too much is a great way to turn a readable 100 line algorithm into a 250 line clusterfuck spread across 16 files.
ok, so you're talking about overdoing it. It's still a good approach when done right.
If code is running once in order, there's no reason to break it up into functions and take it out of execution order. That's just stupid.
Basically if it works and you don't have to touch it to change it, leave it alone.
It was back when a bunch of social networks released app platforms after Facebook's success. When hi5 released their platform, rather than refactoring for our codebase to work on multiple social networks... someone ended up just copying the whole fucking thing and did a global rename of Facebook to Hi5.
For the 3rd social network I refactored our Facebook codebase to work with as many as we wanted. But we never reigned in Hi5, because it had diverged dramatically since the copy. So we basically had two completely separate codebases: one that handled hi5, and one that had been refactored to be able to handle everything else (facebook, bebo, myspace, etc)
Maybe we'll have 'Forest Software' in some time. 'A code forest is an ecosystem where the population of bugs, slugs, trees and weed balance themselves, requiring very little input from the engineer'.
That makes it sound like the problem is more of a spaghetti mess than duplication.
But I think the advice to copy something when you need two versions is supposed to be applied to specific functions or blocks or types. Not entire files. Then it wouldn't have duplicated the geographical code.
It's also important to have a good answer to how you'll identify duplicated bugs. I'm not sure how to best handle that.
This is a terrible advice. According to this, a classic program that loads data from a file, processes it, then writes the results to another file should be a single giant main() that mixes input parsing, computation and output formatting. Assuming file formats don't change, all of those would be used only once. CS 101 style. :D
The primary reason for building abstractions is not removing redundancy (DRY) nor allowing big changes, but making things simpler to reason about.
It is way simpler to analyze a program that separates input parsing from processing from output formatting. Such separation is valuable even if you don't plan to ever change the data formats. Flexibility is just added bonus.
If the implementation complexity (the "how") is a lot higher than the interface (the "what") then hiding such complexity behind an abstraction is likely a good idea, regardless of the number of uses or different implementations.
I drop in to Java shops from time to time, and am more than happy to port my simple class structures that make sense and do things into the 18 level hierarchies described in the article. I just assume there is somebody there that is really invested in all those interfaces, adapters, and impls and I’m not here to start silly fights with them. The code will still work no matter how many pieces it’s cut into and how many unnecessary redirections you add so no worries.
But for my own stuff I like to keep things compact and readable.
And why go so extreme? A main() calling into 3 functions like load, process and save will be still plenty better than a single blob of IO mixed with computation and would still contain no boilerplate.
> I drop in to Java shops from time to time, and am more than happy to port my simple class structures that make sense and do things into the 18 level hierarchies described in the article.
I certainly agree with that, but that has nothing to do with abstraction. Abstraction and indirection are different things. Those terrible FizzBuzz Enterprise like hierarchies are typically a mixture of insufficient abstraction and far too much of indirection. Abstraction reduces complexity, while indirection increases it. AbstractFactoryOfFactoryOfProblems is indirection, not abstraction, contrary to what the name suggests.
I wouldn’t. I’d break it up the same as you, with those 3 functions. After we’d shown we were going to be doing lots of similar things. But given the choice between too complicated and too simple, that’s the direction I’d lean.
Apologies if that wasn’t clear in context.
Sure. But a main() ordered into loading, processing, and saving would be similar amounts of better, despite not using the abstraction of functions.
Unless I'm the one using words weirdly here.
In order to organize code that way, you need to establish e.g. some data structures to represent input and output that are generic enough that they don't depend on the actual input/output formatting. There you have the abstraction.
The key thing is to be able to understand the processing code without the need to constantly think the data came from CSV delimited by semicolons. ;)
>
> This is a terrible advice. According to this, a classic program that loads data from a file, processes it, then writes the results to another file should be a single giant main() that mixes input parsing, computation and output formatting. Assuming file formats don't change, all of those would be used only once.
I broadly agree with you, but devils advocate time: not all abstractions are at the same level.
Writing a static function `slurp()` that takes in a filename and returns the file contents isn't an abstraction in the same sense as having a `FILE *` type that the caller cannot look into which functions like `fprintf()` and `fscanf()` use to operate on files.
I think an opaque datatypes (like `FILE`) are "more abstract" than static functions defined in the same file you are currently reading.
IOW, "Abstraction" is not a binary condition, it is a spectrum from full transparency to full opacity.
Static functions in C would be full transparency (no abstraction at all).
Opaque datatypes in C would be full opacity (no visibility into the datatype's fields unless you have the sources, which you may not have).
C++ classes would be something in-between (the private fields are visible to the human reading the header).
Also, the simpler the interface, usually the more contexts it can be used in. So those abstractions with nice interfaces naturally tend to be more reusable. But I argue this is the consequence, not the primary reason. You probably won't end up with good abstractions by mercilessly applying DRY.
Yes, if the program will be written and tested exactly once, with no change requests to come later, it's perfectly fine to write it as one big main().
It all depends on what the stakeholders need, clear communication with them is the real trick.
If the main was split into well defined, separate pieces, at least you could quickly rule out quite a lot of complexity. If it crashed in parsing, so wouldn't need to understand the processing logic, etc.
Sure it is easy to read one blob of code, if it is only 100 lines of code. But it is a different story if it is 10000 lines and now you have to figure out which of the 100 variables are responsible for keeping the state of the input parser and which are responsible for "business logic".
I mean if it's only like 50 short lines, that would be okay-ish, but in this case why do it in C and not use perl or awk?(i suppose you want fast text processing, so I won't suggest python). If the processing is hard, then you will need debugging (which is better in segregated functions) and to prototype a bit (unless I'm the only one who does that?).
In my mind, the common emphasis on the DRY/WET thing with abstractions leads many people to miss the point of abstractions. They’re not about eliminating repetition or removing work, they’re about making the work a better fit for the problem. Code elimination is a common byproduct of abstractions, but occasionally the opposite may happen to.
I see an abstraction as being comprised of a model and a transformation. The villain isn’t premature abstractions, it’s abstractions where the model is no better (or worse!) for the problem than what’s being abstracted over.
I would add, though, that in my experience you can often identity parts of a design that are more likely to change than others (for example, due to “known unknowns”).
I’ve used microservices to solve this problem in the past. Write a service that does what you know today, and rewrite it tomorrow when you know more. The first step helps you identify the interfaces, the second step lets you improve the logic.
In my experience this approach gives you a good trade off between minimal abstraction and maximum flexibility.
(Of course lots of people pooh-pooh microservices as adding a bunch of complexity, but that hasn’t been my experience at all - quite the opposite in fact)
The same criticisms for microservices (claims that it adds complexity, or too many pieces) are also seen for OOP.
Curiously, while folks sometimes complain about breaking up a system into smaller microservices or smaller classes, nobody every complains about being asked to break up an essay into paragraphs.
Splitting material into individual books makes a little more sense as a metaphor, especially if it's not a linear series of books. You can't just split a mega-book into chunks. Each book needs to be somewhat freestanding. Between books, there is an additional purchasing decision introduced. The end of one book must convince you to go buy the next book, which must have an interesting cover and introduction so that you actually buy it. It might need to recap material in a previous book or duplicate material that occurs elsewhere non-linearly.
A new book has an expected cost and length. We expect to pay 5-20 dollars for a few hundred pages of paperback to read for many hours. We wouldn't want to pay cents for a few pages at a time every 5 minutes. (or if we did, it would require significantly different distribution like ereaders with micropayments or advertising). Some books are produced as serials and come with tradeoffs like a proliferation of chapters and a story that keeps on going.
Anyway, it's a very long way to say that some splitting is merely style, some splitting has deeper implications, the splits can be too big or too small, and some things might not need splits at all.
[author] uses the [simile] to argue the [argument].
The obvious flaw in the [argument] is of course [counterargument].
[quote]: Curiously, while folks sometimes complain about breaking up a system into smaller microservices or smaller classes, nobody every complains about being asked to break up an essay into paragraphs.
[author]: Mr_P
[simile]: microservices or smaller classes are like paragraphs in an essay.
[argument]: since no one complains about breaking up an essay into paragraphs, no one should complain about breaking up a system into paragraphs.
[counterargument]: breaking up a system in smaller microservices or classes is not at all like breaking up an essay into paragraphs, which I think this comment has demonstrated.
There are orders of magnitude different amounts of work in each of these cases. (I’m not saying it’s a lot of work but it’s still significantly more in some of those cases relative to the others.)
They would if each paragraph of that essay lived at a different domain/url.
> If something is used once, ignore any abstractions. If it's used twice, just copy it, it's better. If it's used 3 or more times, look at writing an abstraction
My rule of thumb is that there are only three quantities in the software development industry: 0, 1 and infinity. If I have more than 1 of something, I support (a reasonable approximation of) infinite quantities of that something.
The right word is "generalization", and that's what you are actually doing: you start with a down-to-earth, "solve the problem you've got!" approach, and then when something similar comes up you generalize your first solution.
Perhaps part of the problem is that in OO, inheritance is usually promoting the opposite: you have a base class and then you specialize it. So the base class has to be "abstract" from day one, especially if you are a true follower of the Open Close Principle. I don't know about others, but for me abstractions are not divine revelations. I can only build an abstraction from a collection of cases that exhibit similarities. Abstracting from one real case and imaginary cases is more like "fabulation" than "abstraction".
The opposite cult is "plan to throw away one", except more than just one. Not very eco-friendly, some might say; it does not looks good at all when you are used to spend days writing abstractions, writing implementations, debugging them, and testing them. That's a hassle but at least once you are done, you can comfort yourself with the idea that you can just extend it... Hopefully. Provided the new feature (that your salesman just sold without asking if you could do it, pretending they thought your product did that already) is "compatible" with your design.
The one thing people may not know is how much faster, smaller and better the simpler design is. Simple is not that easy in unexpected ways. In my experience, "future proofing" and other habitual ways of doing things can be deeply embedded in your brain. You have to hunt them down. Simplifying feels to me like playing Tetris: a new simplification idea falls down, which removes two lines, and then you can remove one more line with the next simplification, etc.
A specific example is getters and setters for class variables. If another class directly accesses a variable, you have to change both classes to replace direct access with methods that do additional work. In other languages (Python specifically), you can change the callee so that direct access gets delegated to specific functions, and the caller doesn't have to care about that refactor.
Edit: I suppose it may be argued that you need to set some other state when you set a member variable. If that's the case, then it's no longer a getter or a setter and the function should be treated differently.
You can say that's not a getter/setter, but then your definition is just different than the people you're responding to.
interface Store { Query() }
// these all have the Query() method
type/class MySQL implements Store
type/class Cache implements Store
type/class Logger implements Store
var db Store
db = new Logger(new Cache(new MySQL()))
Of course, in more limited languages/environments they're probably the only tool you have, so there's that.
I get they can be cumbersome, but using them really matters especially as a project grows... an API that has a simple single client today may have many different (and concurrent!) ones tomorrow. The pain of using S&Gs now saves refactoring later.
At what point is it cheaper to just refactor into getters/setters later when needed? That point _has_ to be miles behind me.
Another problem (from a class/library-consumer point of view) is having getters/setters suddenly becoming more expensive to call, blocking, or even having side effects after an update.
It often only affect the runtime behavior of the code.
Changing the interface, however, will give me a hit that something else has changed.
That isn't the only type of OO. Look at CLOS in Common Lisp for a counterexample: https://wiki.c2.com/?HowObjectOrientedIsClos
Maybe for debugging you want to log a callstack every time the field gets accessed, for example.
Or when you set the field, you should invalidate some cached value that uses it.
That is just as bad as a general rule as "What if it ever changes, we need to abstract over it!". As always: It depends. If the abstraction to build is very simple, like making a magic number a named variable, which is threaded through some function calls, at the same time making things more readable, then I will rather do that, than copying copy and introducing the chance of introducing bugs in the future, by only updating one place. If the abstraction requires me to introduce 2 new design patterns to the code, which are only used in this one case ... well, yes, I would rather make a new function or object or class or whatever have you. Or I would think about my over all design and try to find a better one.
Generally, if one finds oneself in a situation, where one seems to be nudged towards duplicating anything, one should think about the general approach to the problem and whether the approach and the design of the solution are good. One should ask oneself: Why is it, that I cannot reuse part of my program? Why do I have to make a copy, to implement this feature? What is the changing aspect inside the copy? These questions will often lead to a better design, which might avoid further abstraction for the feature in question and might reflect the reality better or even in a simpler way.
This is similar in a way to starting to program inside configuration files (only possible in some formats). Generally it should not be done and a declarative description of the configuration should be found, on top of which a program can make appropriate decisions.
The key question is whether there is a functional dependency or just a similarity between some lines of code. If there is a functional dependency, it should be modeled as such the first time it is repeated. If there is only coincidental similarity then introducing a dependency is simply incorrect, regardless of how often any code happens to get repeated.
As others have said, this is a good rule of thumb in many cases because finding good abstractions is hard and so we often achieve code re-use through bad abstractions.
But really good abstractions add clarity to the code.
And thus, a good abstraction may be worth using when there is only two, or even just once instance of something.
If an abstraction causes a loss of clarity, developers should try to think if they can structure it better.
EDIT: This comment below talks about good example of how a good abstraction adds clarity, while a bad abstraction takes it away: https://news.ycombinator.com/item?id=31476408
If you write an integration test, and it fails, what's broken?
That's a valid concern, but if your unit-tests are only for making sure that part you just wrote works as expected than just have a test-case up for that specific part, and change it when you move to the next part.
The value of unit-tests is supposed to be regression testing: when you change something that breaks a different unit in a different part of the stack.
> If you write an integration test, and it fails, what's broken?
Well, I debug it it the same way I debug any bug. After all, most bug reports are from a full execution in the field; I am probably already set up to debug the full application anyway[1].
[1] Once a bug reproduction is set up in a fairly automated way.
A very good one, with effective use of matchers, you don't even have to read the test to know what you did wrong. You just know from the error message what you broke.
>
> A very good one, with effective use of matchers, you don't even have to read the test to know what you did wrong. You just know from the error message what you broke.
Agree, and agreed. The counterpoint is that unit-tests take time to write and time to maintain - you have to balance that time spent against the time that you would spend debugging an integration test.
If your unit tests are hard, you need to refactor your code.
So? You're going to have them anyway or else you can't deploy.
> Major functionality changes can affect all of your tests. With unit tests they may invalidate a few, but that’s okay because they were cheap to begin with.
If you start with unit tests, then the integration tests are just verifying the plumbing. That only changes when the architecture changes, which is hopefully a lot less than how often the requirements change substantially.
The logic that usually gets ignored in unit tests is the ones that actually needs to be tested, but skipped because it is too difficult and might involve a few trips to the database which makes it tricky (in some scenarios you need valid data to get a valid test result, but you cannot just go grab a copy of production data to run some test).
And then there is the problem of testing related code, packages and artifacts being deployed to production which is really gross in my mind and bloats everything further.
A team I've worked on has resorted to building actual endpoints to trigger test code that live alongside other normal code (basically not a testing framework), so that they could trigger test and "prove the system works" by testing against production data at runtime.
>The reason a lot of Java or C# code is written with all these abstractions is because it aids unit testing.
That is the justification for talking about testing. Code is being ripped apart to make it easier to test, while the tests that are used as a justification for ripping apart the code are low quality as 99% of the work in unit testing is thinking of and setting up the test case, not the actual test code.
The only possible justification for duplicating code would be that creating an appropriate abstraction is harder. Given that there are generally economies of testing when you factor out common code, that's usually just not true.
"Duplication is evil" is a more reliable mantra.
Yes it means stuff needs to be changed in 2 places. Yes it means someone can change one and not the other. But it also means that each thing can be maintained on it's own without worrying about another. In the early stages you don't know how much can truly be reused and whether you're just cornering yourself. I've had scenarios in the past where we've written an abstraction around some common code and then the third application we want to use it in just does not fit the model we initially thought of. Could we change the library? Yes obviously. But are we going to face the same issue on the 4th project to use this library? Probably. It's a large maintainance load. At some point you end up making breaking changes to the library and you're comitted to either maintaining multiple major versions, or maintaining an abstraction that is supposed to work for every scenario, which can be a huge time sink.
There are tradeoffs to be made. I'd rather lose the maintenance burden of library when consumers have vastly different needs and just take the hit if having to do a Sourcegraph search for usages of some code. This search would need to be done to find all consumers of the code anyway if it was a library. So the end result is rarely different in my experience.
It works both ways
Someone modifies code thats used in both places and breaks othef thing
If you are looking for example into System Integration, Data Integration, ETL and so on, not using a canonical format from the beginning, will get you into the type of almost exponential grow in mappings between sources and targets.
https://www.bmc.com/blogs/canonical-data-model/
https://www.enterpriseintegrationpatterns.com/CanonicalDataM...
I do agree a lot of abstractions in C#/Java seems to be testing implementation stuff leaking into the abstraction layer. A lot of inversion of control in these languages seems purely to allow unit testing, which is kind of crazy.
Personally I prefer the "write everything in as functional a style as possible, then you'll need less IoC/DI". This can be done in C# and Java too, especially the modern versions.
Once is an incident. Deal with it.
Twice is a co-incident. Deal with it. But keep an eye out for it...
Third time? Ok, this needs properly sorting out.
This online chat had two functionalities, chat with a worker a and leave a message from worker with suggestions as to what to look at in response, there was no connection between these two functionalities specified and so my friend had written it without connection, it was difficult without doing a full rewrite to get state information from one part of the application to another one (this was written in JQuery)
Anyway, 6 months+ down the line it got respecified, now it needed shared state between the two parts of the application, which meant either significant rewrite or hacks, so hacks was chosen. Ugly hacks but worked (I think ugly hacks was the correct choice here definitely because chat application almost completely scrapped a year later for a bot)
After I was done I said "but why write it like that?" It was specified no state was needed between the two parts, "yeah but it should be obvious that is going to change, they would keep wanting to add functionality to it and probably share state as to the two communications channels"
tldr: there are some potential changes that seem more likely than others, and the architecture should take those potential changes into consideration.
The reason a lot of Java or C# code is written with all these abstractions is because it aids unit testing. But I've come to love just doing integration testing. I still use unit testing to test complex logic, but things like "does this struct mapper work correctly" are ignored, we'll find out from our integration tests. If our integration tests work, we've fulfilled our part of the contract, that's all we care about. Focus on writing them and making them fast and easy to run. It's virtually no different to unit tests but just 10x easier to maintain.