Ask HN: When do you use a library vs. implement something yourself?
At one end are single-function npm libraries that led to the left-pad fiasco[1].
At another end are things that you should only be implementing yourself in highly unusual circumstances, such as cryptography libraries.
Where on this spectrum do you usually implement the thing yourself, and what is your thought process?
What are the costs of importing a library, and how are those costs different for different projects? Are there security risks? What are the benefits?
Conversely, what are the costs and benefits of implementing something in-house?
Has there ever been a case where you've tried to roll your own and it ended in disaster? Or where you'd wished you'd never added a dependency on some library?
[1] https://www.theregister.co.uk/2016/03/23/npm_left_pad_chaos/
(
69 comments
| 72 points
| on HN
)
1. It is still alive
2. It is on GitHub so I can easily read the source and look for issues
3. People star it
4. It has traction ie >10k downloads preferably >50k
5. It occupies a name that makes sense ie I’d rather trust using “redux-immutable” than “steel-porcupine”, since it unofficially lends itself to the “vetted one true solution”
6. Then I look at the source quickly and make sure it looks well done under the hood
7. Finally, if it’s particularly confusing, I might run a quick google check to see if there’s blogs about it on the internet explaining how to use the advanced features and check the date on the blogs (1-2 years ago = good, it’s mature)
If any of the above are too dicey and there’s no good options, I put the relevant code in a top level folder called “modules” (or otherwise) with the intention that in the future I might get to delete it if the package ecosystem gets more mature in that space. Sometimes I even get to polish those files up and actually publish them.
1. Your standard library can do it. Whatever you do, RTFM, learn your language inside out and prefer the standard library for whatever you do, even if it looks a little worse — especially if it's not more than 50% more code than using a library for convencience functions. This already goes 80% of the way to prevent "left-pad"-like disasters.
2. Everything that is your core business logic. Don't succumb to the fascination of rule engines and state machine builders. Just write that damn code yourself, future you will be way happier for it.
3. If you're sure that you'll never need more than 10-20% of a library, but eventually will need something considerably different starting from there. This may be grounds for justifying building your own one. If you do that and it doesn't give your business a unique edge, consider open sourcing it from the start.
4. If in doubt, rather stay away from big, fat frameworks until they give you a really big edge (like in Frontend development) or you tend to do a lot of rinse-and-repeat stuff. They tend to solve a lot of your problems, but rot fast and you have to continuously keep up with the Kardashians.
left-pad is a red herring - the problem there was poor dependency management, not the use of the library itself.
> 4. It has traction ie >10k downloads preferably >50k
Why ? Your home-made code probably don't check those boxes.
An unmaintained library is usually more difficult to fix because fixes are no longer committed back or released. At that point you can choose to vendor and maintain your own copy, or replace it with something else. And at either of those points you’re no longer in context.
My point wasn’t really geared at debugging a piece of code. A library, my own, or a coworkers are roughly comparable. The cost to fix is a lot higher in a library though.
It’s not a universal rule, despite the way I originally phrased it.
On the flip side, I have found libraries that are unmaintained and are pleasant enough to understand. I've forked those and used them, happy that:
1. I can quickly fix any bugs that I face
2. Someone else can make use of my maintenance.
The amount of people who uses a product does not determine is quality.
However stars can determine to an extend whether it's a project worth looking at or not.
A mature project that just works does not need new commits.
- If you file a bug report does the bug get fixed? Do you even get a response?
- If you ask for help from the community, do you get it?
Not necessarily whether there are commits or not.
If I really need to use some library (for example Graphic toolkits etc.) I tend to try to understand it by looking at its source. I try to make some modifications in it, try to make them run. That way I get somewhat an understanding of the library and the problem domain itself.
Usually you only need a small part of a library. You can look at the implementation at that library and then incorporate it in your program and slowly improve it. That way your dependencies won't get bloated, you'll get a customized solution with little effort and if you encounter problems you'll know where to fix them. That's a trick I've learned over the years.
https://msdn.microsoft.com/en-us/library/system.string.padle...
For example, Go, C and C++ have bad dependency management, so most programmers will try to stick to stdlib solutions, only moving to dependencies if copy-paste is an overkill (I wouldn't copy-paste sqlite wrappers, but I would copy-paste a left-pad like function).
NPM and Rust (which isn't managed) has great dependency management, so they move things out of stdlib into packages.
But, it backfired when no one is sure which packages are official, so Rust made a "nursery" of officially maintained out-of-stdlib standard functions.
Example: I need to parse excel-files in python => straight to looking for a library, I don't even want to start looking into excel-format specs.
Example2: I need to parse JSON in Java. First look if Java have this natively (I don't write too much Java) and then go look for a lib. Others must have had this problem, and they most likely solved it as good as or better than me - the surely devoted more time to it, if the bothered to publish a lib.
I review if the library project seems alive and being updated. I review if it is super easy to use. Then I include it, no more fuss. Most time I end up checking out 2-4 libs to compare ease-of-use and liveliness etc.
I admit that I could do a lot more on security review. Most of the time I fallback to the thinking that this is OSS and/or a widely used lib, and if there was a security issue that I would be able to find, others would have found it by now.
*: What too long is varies I guess, but you always have a gut feeling of how troublesome something is to do. Maybe compared to the value of x feature.
Thats how it is for most all hobby projects. If it is projects for a client, I worry a bit more about security, but the primary driver for decisions are that others must have solved the same problem and most likely better.
And of course in some organisations you'll find it-policies, security policies, dev guidelines etc. that you just need to follow. Then you do that - the client is paying for slow then, by their own choice.
If I want some functionality that's difficult to implement, I'll see about including it verbatim in some branch of the source tree.
If it's a large amount of code, or not license-compatible, or needs to be updated apart from the program, then I'll resort to using a library.
Library updates are the number one reason for code rot, in my opinion. More dependencies is more chance for pain later. I try to pick mature platforms and lower-level libraries which change less frequently.
Is this a utility-like function for you, or is it defining competitive advantage for your project? Do you need to own the roadmap for this functionality, or are you happy for others to define the roadmap for you? Are you genuinely innovating in this functional or non-functional area - if not, why are you spending time/money/effort on it? What is the TCO of the given solution vs building and maintaining the functionality yourself?
As developers we think the solution to any problem is to develop. It often isn't. If you need something to serve stuff off port 80 at scale, would you build a web server, or install one? If you need crypto, same question.
Spend your time where it counts, be grateful for the decades of work that has gone into the stuff you can take for free out there, use it wisely.
While I try to avoid premature abstraction, on libs that have high volatility or otherwise don't score very high, I'll put a thin interface layer between my use and the true impl. This saved me recently as I had a Scala trait abstract the async Postgres client lib I was using, then I hopped from one to another to another over a couple of years until just recently writing one because the others sucked for various reasons. Just make sure you have integration tests and you'll feel comfy in your choice.
With more complex systems, how much integration/customisation is needed vs building something yourself?
Often I'll look for a library, find a few candidates that don't really suit well and 'resort' to committing to writing something myself. Similar to planning a project, it can take time to work out how much time something is going to take.
- how value adding is this functionality?
- how well does the library meet my need?
- is the library still under active development?
- how stable is the library's API? (Building against an unstable API becomes a tedious waste of resources, fast.)
- how good is the documentation?
- is it popular/well-used? If so, this makes it easier to get help on Stackoverflow, or wherever, when I have a problem.
There are times where we've started out with a library, but moved to a homegrown implementation, or vice versa.
For example, we work with a lot of CSV input data. We started out with a library, because properly parsing CSVs is, well, not exacly tricky, but there are a pile of nuances and edge-cases you need to take account of. It was just much easier to use a library for this so that we could focus on adding business value elsewhere.
Anyway, the size of the files we needed to work with grew, so we swapped the original library out for a different, faster, library.
That worked well for a while, but then the size of the files we needed to work with grew again, and the library was too slow, so we swapped in our own faster CSV reader implementation for working with these large files.
We still use the library for other CSVs because our implementation makes certain assumptions about the content of the files that don't hold for the smaller files we work with.
The obvious place to develop a library inhouse is when there is no compatible third party solution available.
The obvious place to use a library is when one is available that a) Implements a feature that would requires weeks of work b) implements the specific feature one requires c) on the technology platform of the product d) is compatible financially and licensing wise
If a,b,c and d hold then third party library is a pretty good option. Depending on the complexity of the problem one might be really lenient on c) and do whatever to integrate a MagicalPixieDustGenerator no matter how badly it fits the current tech stack.
Code reuse is one of the silliest arguments ever for librarization. It's totally fine to replicate few lines of code, it's just typing. If you don't have tests to verify functionality you are screwed anyway.
(2) Another experience I had was when I started the SageMath project. I began with the plan to look at other code and implement everything myself, rather than use existing libraries. That lasted about an hour, since it was way, way too much work (to put it mildly). So I switched to using Python as the user language, then using as much as possible existing C/C++ libraries (like PARI/GP and GAP and Singular) to provide functionality. The result was again a combination of using libraries but also writing a new library.
Long story short we implement something ourself if:
1.It is a differentiator for our company because 2.
2.We can make it better than available libs and 3.
3.Making it better represents a real change on our company or the world, changes like "It will make possible what was not possible" or "will halve the cost of X, because Y".
4.We need to parallelize-make it on hardware.
Implementing something inside is extremely expensive as requires salaries of highly paid people, makes those people unavailable for doing other things, and takes always more time than planned.
On the other hand with your own code you could do magic. You can hardware accelerate(with electronics and FPGAS) it to use 10.000, 50.000 less energy, or GPUs to make it 100-200 faster. You control everything to suit your needs.
We normally start using standard libraries and replace then slowly with our own code if it will make a difference.
We do the same with parallelization: we start with multiCPUs, then GPUs, then FPGAS. Then we can even think on offering investors the option of making ASICS if functionality is proven and very well tested.
Disasters? Yes of course. It is part of experience, they happen like great successes. When you risk you risk disaster, but you give yourself the option to success greatly too.
When I was a kid I learned to ski probably faster than anyone around. But it was because I did not play safe like most did and keep falling(at great speed) until I did not.
In life you could do the same.
1. It would take a significant time (> a few days) to write yourself.
2. The library is widely used.
3. The library has a history of supporting major versions, you don't want breaking changes to come with security and minor fixes.
4. The library is open source. Not always a complete deal breaker, but you need to think of long term costs.
5. You're going to use a significant chunk of the library, don't import a big complicated library for a single function.
What works best for you might not be the correct answer for someone else.
For us, we initially used libraries for all lat/lon and distance calculations and then refactored to move the functions to our internal code and get rid of the libraries. For algorithms like DBSCAN , even if it's straightforward, it didnt make sense to re-implement it.
I guess it boiled down to the effort required to move inhouse vs the benefits
Beyond that, if it's not core business logic, it gets downloaded.
At one end is a client wanting a quick mock up - yes use libraries.
At the other end is a surgically implanted heart monitor - please dont use a library that depends on an API call.
Using libraries certainly gets the job done quicker (in the short term), but it adds technical debt and brittleness to any project.
Javascript build my own as they are not.
(Joke with a hint of a more serious problem).
I like this metric because it means I'm saving time just writing the same thing I would have written anyway.
But that is not the rule i work after, It just seams like that is how i work when i work now that i think about it.
For everything else usually I have a simple flowchart: is it a core feature and a businnes advantage of the platform? Implement. Otherwise, if exists a library, use the library.
Personally, I try to make an informed gut-feel decision about the short term and long term benefits of pulling in a library for anything. The main inputs to the decision are how difficult rolling my own would be, whether there are any complex risks (such as security implications I'm not well versed to handle, or other such topics), and how much use / mileage we will get out of the dependency. Very often this comes down to whether you've done something similar before, and are confident that you can handle it well enough. I think I tend to lean towards rolling my own, or "NIH syndrome" a bit more than the average developer, but this is for a few reasons. Take this with a pinch of salt, and for context, I would of course agree with not rolling your own crypto, etc - I'm mainly referring to the gray area in the middle.
1) It's usually quite difficult to find a library that nails the level of complexity vs pay-off that I want in a solution. Most of the time, libraries that I find are a complete overkill for my usage, or have large surface areas that I don't need / want. If I can roll a custom implementation in under 50 lines of code, I will generally do so.
2) While you often get immediate payoff for pulling in a dependency, that payoff very often turns around and becomes a net loss of time the moment you run into a complex problem. Pad-left is a silly example, but I think we've all encountered situations where we pulled in some library only to run into a strange edge case. Having wasted a day, you're then thinking "I honestly could have done this better, given a few hours".
3) This point tends to depend a lot on the exact environment you're working with, but managing dependencies can be pretty painful. As a simple example, I've had a number of Angular and Ionic projects with a large number of dependencies become extremely non-trivial to upgrade. If you're talking about a small number of such projects which are frequently worked on (say, flagship apps of a startup), I suspect this is not much of an issue. When you're dealing with dozens of apps for a large corporate, some of which may only get attention in a year or so, this can be a painful burden. A hilarious parody of pad-left is "is-thirteen" [1], which drives the point home for me.
4) Rolling your own also tends to architecturally firewall your implementation (if you've done it right). I.e. it is easier to take your custom implementation and just point it at a 3rd party library, than it is to migrate from library A to library B, if you haven't firewalled them nicely.
The number of times I've regretted rolling my own has been far outweighed by the number of times I've pulled in the wrong dependency.
In summary or as examples, I would never willingly pull in "pad-left", would always stick with a well known UI framework rather than roll my own (primarily for various security reasons), but have written simple templating engines, markdown parsers, etc, without regret. On the other end, I would never roll own crypto, and would be highly hesitant to even come up with my own crypto schemes involving well-implemented crypto, as those can be just as weak.
[1] https://github.com/jezen/is-thirteen#is-thirteen
* Reusing someone else's implementation gives you the benefit of an initial kickstart. It also can give you future benefits: a library may contain improvements that you may need in the future, at which point you get them for free. Infinite productivity!
* Building your own implementation has an initial cost, but it reduces your maintenance burden to only the features you actually use. It also insulates you from counterparties whose interests may not align exactly with your own, or who accidentally introduce security risks.
If only we could magically unbundle the benefits of other people's code from their limitations, have features magically appear when we need them, and be magically robust to security holes in features we don't use. Here's the closest I've been able to get so far:
Use libraries by copying them.
Let's see why this is good. For starters, it makes you robust to the leftpad failure mode. The library is literally copied into your repository. Who cares how small it is anymore?! It could literally vanish off the face of the rest of the earth, and your code would continue to work.
A less obvious benefit is that copying a library puts you on a trajectory of taking responsibility for your dependencies. You don't need to understand all of it on day 1, but as you use it and run into issues you should be gradually understanding its internals better, gradually resolving features that you use, and features that you don't. That then gives you the option to rip out stuff you're sure you'll never need. Or to rip out stuff you may need in future, but is easy to put back (this is analogous to deleting code without fear because version control).
I consider an example of exemplary library use to be how I copied the termbox library into Mu (https://github.com/akkartik/mu/commit/5f1285238b), periodically merged commits from upstream (https://github.com/akkartik/mu/commit/9ba313ab7f), gradually cleaned it up to fit better with my project (https://github.com/akkartik/mu/commit/9a31c34f0f), and gradually stripped out code from it that Mu does not require (https://github.com/akkartik/mu/commit/c04baba4f2; https://github.com/akkartik/mu/commit/8e7827dfcf; https://github.com/akkartik/mu/commit/547ec78bf2). In the process I made some wrong turns, deleting features that I later decided I wanted (https://github.com/akkartik/mu/commit/10a3b8cca2) and created bugs for myself (https://github.com/akkartik/mu/commit/3315a7d3bb; https://github.com/akkartik/mu/commit/0c0d1ea5cd). But when it did things I didn't want, I was now empowered to change them (https://github.com/akkartik/mu/commit/ee1a18f050). One of my patches was useful upstream, so I submitted it: https://github.com/nsf/termbox/commit/0730826a07. I would be in no position to submit that patch if I hadn't taken the trouble to understand termbox's internals. And that shows a final benefit of copying and privately forking libraries: it makes you a better citizen of the open source world, because open source depends on eyeballs, and using a library blindly helps nobody except your (extremely short-term) self.
Now you may say, this sounds like a lot of work. I can't do it for all the libraries I use. Start small. Pick one dependency that is pretty central to your project, bring it in. See how you like it. As you gain experience come tell me what you discover. (My email is in my profile.) I didn't have all this thought out when I copied termbox into Mu. Benefits gradually became apparent.
[1] http://akkartik.name/post/libraries
[2] http://akkartik.name/post/libraries2
(Past versions of this comment: http://arclanguage.org/item?id=20213; https://lobste.rs/s/w4auk6/more_shell_less_egg_all_this#c_nv...)
* The code is the best in the world and would take anyone years to compete with its speed. Libraries like gmp, fftw etc come to mind.
* The API is good. A good API extends the language in a way that makes your code better and more enjoyable to write.
I usually try to avoid dependencies where possible because it will always make life easier later on. Introducing a dependency for a single function should be a sackable offence imo. At that point you're just too lazy or incompetent to do your job.
I'm suppose to make sure stupid shit like that doesn't happen. It's not a waste of my time. It's my job.
The whole leftPad controverse makes a pretty good case for rewriting. It would have saved many people a lot of trouble.
Also, if you know your langauge, rewriting it is probably faster than finding, installing and using a package.
As far as I remember, it's always the other way around. But I have enough sense to not implement my own cryptography library and use it in production.
> Or where you'd wished you'd never added a dependency on some library?
That happens a lot, or more frequently it's some library already being used that is overcomplicated and won't work with something else I need it to work, like a build tool, or is in a cross platform language like node but doesn't work on one particular, major OS. But when this happens it's maybe never the case that it would have been reasonable to write it myself. I just have to find a less shitty alternative library.
In general, it's blatantly obvious when you need to do which, you have more much bigger other problems, the chance of this issue even popping up is pretty low, and the resolution is probably fairly straightforward.
For every library you use, you are now trusting everyone that has publish rights to that library and all of its transitive dependencies, so libraries aren't free.
Also, sheepishly, it's so easy to publish on NPM that I'll sometimes just re-implement a library with my creature features and publish it instead of making a case for them in the existing library's issues. Awful, I know.