"This'll take an afternoon" - three weeks later......
Programmers are notorious for this.
BUT even apart from this problem ... you absolutely should use every dependency you can that will save you time.
Try to write less code not more. When you write code you write bugs, add complexity, add scope increase need for testing, increase the cognitive load required to comprehend the software, introduce the need for documentation..... there's a vast array of reason to use existing code even if you truly could estimate it and build it in an afternoon.
You also assume that you understand all the edge cases and fickle aspects of the dependency, all the weird ins and outs that the dependency author probably spent much resources understanding, fixing and bug hunting.
There's a hard fact that proves the above poster to be wrong..... how many dependencies took only an afternoon of time in total to write? Hard to say (maybe look at the github commit history) but I'd guess almost none. It didn't take the dependency author an afternoon, so why will it take you an afternoon?
Even worse .... you just lost an afternoon coding features for your core application.
Multiply this by every dependency that "you could build in an afternoon" and you'll be in Duke Nukem Forever territory.
I'd advise doing the opposite of this articles suggestion.
Find a dependency that will save you an afternoon? Grab it.
Dependencies have costs:
- Dependencies break over time. They have a nonzero maintenance cost.
- They impose API boundaries on you that may not fit your existing data structures
- It's harder to change underlying bugs
- They might introduce security issues
Sure, use dependencies. But there's a reasonable position between "never write any code" and "never take on dependencies". Of which NPM is one of the only ecosystems being at one extreme.
Every single project I go into seems to have a smorgasboard of dependencies, then when I take the time to investigate one of them I find out it's being used incorrectly by at least 50% of the team because they don't even understand how they work at the most basic level. Which is pretty understandable because by the time anyone gets through understanding 10 of the 85, they've probably been kicked off the team for not actually building anything.
People love to say rubbish like "write less code!", as if LoC is the only metric that matters (weren't we past that thought process by the 90s?). Which goes a long way to explaining all the fucking terrible codebases I have to work with where it's impossible to accomplish anything without reading documentation for 8 hours, when it would take 20 minutes to just read even a semi-readable piece of code that implements whatever requirements you need from the dependency.
On a C code project for a large Fortune 100 company a half dozen years ago, I encountered a pesky header include that made no sense. And that header was part of a patch that I really did not want to pick up, so I started digging into it.
Turns out that they had some constant in the code, and the developer just did a grep for that value in the source tree, and that constant already existed in an existing header file, so they just included it.
And that CONSTANT_VALUE_STRING had nothing to do with the technology that the C source was addressing. So some lazy slacker pulled in a random header file that contained the proper constant value for an unrelated technology.
The dependency on that was pure lunacy on so many levels.
And that was an internal dependency, not an external library.
So the lesson here? Not all dangerous dependencies are external.
Or two jobs ago where two developers who had worked at the company for 10 and 15 years respectively were maintaining a bespoke 15 year old EHR written in PowerBuilder and depended on SQL Server 2003 - in 2016.
Every company thinks they are their own special snowflake where cross cutting concerns can’t be handled by a third party.
I suppose because it was easier than working in the actual problem domain, which they knew little about and didn’t care to learn.
Ya got me ... pypi, crates, NPM, gems, cpan I'll use em all! Shhhh.... if my employer finds out they'll fire me.
:-)
I keep hoping things like [1] are a joke but I'm starting to suspect they're not.
It often takes less time to write some code than to understand someone else's code.
Most programmers I've worked with get lost easily when jumping through layers of other people's code. I certainly do.
Solid, well tested dependencies that solve hard problems are worthwhile. But dependencies have a cost in debuggability and maintenance, so it's worth using them with care. And often, they aren't worth the time, when compared to writing a dozen lines of code.
But conceding that charitable assumption to the article, I agree with its basic premise: dependencies cost a lot of time in diffuse, non-codey ways.
There are AAA dependencies you pull into every project, but most other dependencies require a good degree of due diligence, evaluation, risk, and their own long-term maintainance.
Its not that it always tips the scales all the way to 'roll your own', but I think the cost of new dependencies is underrated.
So do your upfront work, by all means. It isn't an all or nothing decision.
> Find a dependency that will save you an afternoon? Grab it.
Agree. The point of the article, though, is that dependencies are often saving much less time than they promise - so much less that it's better to avoid them.
From my experience with these personal failings, the problem usually comes from the question being phrased in the context like, "before you begin working on this, how long do you think this will this take you to complete?". If there's no opportunity to scope, with requires not insignificant work towards the solution, the estimates will always be wrong. If I understand the actual scope of the problem, which means have the architecture mostly worked out, and have a bit of experience (and luck), my estimates can be pretty close, usually eaten up by that oh-so-seductive feature creep that ruins my work file balance.
In this day an age, this many years into open source licensing, if your team is not on top of that from day one, they have failed as a team.
I worked at Bell Labs from the mid 1980s to 2000, and by early 1990s (1992? 1993?) they already had a full internal team dedicated to open source licensing issues, including training and consulting. That was 27 or 28 years ago. Before some of the developers on this thread were even born.
Absolutely. As long as it does save you that time over the foreseeable lifetime of the project. Or you are deliberately incurring a technical debt because of some deadline.
On the other hand, saving an afternoon (or even a week), over the next two weeks means very little.
Have you heard the adage about premature optimization being the root of all evil? Yes, even with the second part. What is the premature optimization here, in your opinion?
Most of cases developers create something new - that's the state of industry now, not too good but it's how it is. If you'd be refactoring the existing code - sure, find the problem, design the solution, have reasons going from A to B. If, however, you're writing new functionality, you don't know if you'll have problems of this kind with this code - so optimize for developmentality. You can remove those excessive crutches later - if and when you need them. In my experience, having them trumps looking into code and spending time figuring what it does mere months later - your own code, that is.
If it's not hard to use that library you're probably better off unless it's a problem you understand very well and will see a real advantage to tackling differently. For example, if you use a library and don't like it that experience will still be useful for having clarified what exactly it is that you want to do and the rough size of what you're taking on.
all I needed was
const fs = require('fs');
fs.copyFileSync(process.argv(2), process.args[3]);
- You should absolutely use community-supported tools to solve your problems.
- You should substitute idiomatic code for libraries.
You have made an argument for the latter that does not detract from the former.
Lots of things can go wrong when writing a file: https://danluu.com/deconstruct-files/
So, if I think something looks like about a day’s work, I’ll actually estimate it at about 3.5 or 4 days. Thus, for a project to qualify as “just an afternoon,” I’d have to naively estimate it at under an hour.
I rarely have time to spare, but I also rarely go over by more than maybe a third.
Your multiplier may vary depending on how horrifying your codebase is. On a side project with good test coverage, my multiplier is only about 2.
This is sometimes true, but often not. From SFTP libraries to SVG rendering libraries, there have probably been about 3-5 major dependencies of my company's project that I have had to learn and extend or fix bugs in to make them work just in the last year.
And sometimes this means using our own fork that we have to keep maintained.
I'm not saying I would have rather written these particular dependencies from scratch, but they were definitely not cost free. Nor are they all of better quality than what I would have produced had I written them from scratch.
That's the other common refrain - to "defer to the expertise of the crowd".
Don't get me wrong, many 3rd party libraries are of great quality by amazing men and women who I am very thankful for. But certainly not all of them.
There's no magic that says "every third party library is made by an expert with the highest standards".
var cache = [
'',
' ',
' ',
' ',
' ',
' ',
' ',
' ',
' ',
' '
];
function leftPad (str, len, ch) {
// convert `str` to a `string`
str = str + '';
// `len` is the `pad`'s length now
len = len - str.length;
// doesn't need to pad
if (len <= 0) return str;
// `ch` defaults to `' '`
if (!ch && ch !== 0) ch = ' ';
// convert `ch` to a `string` cuz it could be a number
ch = ch + '';
// cache common use cases
if (ch === ' ' && len < 10) return cache[len] + str;
// `pad` starts with an empty string
var pad = '';
// loop
while (true) {
// add `ch` to `pad` if `len` is odd
if (len & 1) pad += ch;
// divide `len` by 2, ditch the remainder
len >>= 1;
// "double" the `ch` so this operation count grows logarithmically on `len`
// each time `ch` is "doubled", the `len` would need to be "doubled" too
// similar to finding a value in binary search tree, hence O(log(n))
if (len) ch += ch;
// `len` is 0, exit the loop
else break;
}
// pad `str`!
return pad + str;
}The most egregious is the bad big O analysis for a pointless binary search. The loop does indeed run O(log(n)) times but `ch += ch` still takes O(ch.length) which is growing exponentially. It ends up being a complicated way of still taking O(n) time while creating a lot of intermediate strings.
It isn't any faster than just creating the padding with a loop or `new Array(len).fill(ch).join('')` or `ch.repeat(len)`
It's not overengineered if thousands of downstream projects are relying on it, some of which might see significant benefits from those performance optimizations.
For example, it's missing warmup. The results aren't being consumed in a way that wouldn't optimize them away. The framework itself imposes a pretty large amount of overhead (moreso than I'd expect from leftpad).
It is somewhat likely that what they are measuring isn't leftpad performance, but rather how fast the JIT ends up optimizing the benchmark code.
I'd suggest watching this video https://vimeo.com/78900556
It's about microprofiling the JVM, but the principles are the same for other JIT based languages (such as javascript).
Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1].
There's no portion of the code that does warmups. There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. There is a lot of code though... so that's... good?
You make the assumption that just because a lib is popular or widely used it is "correct" or "the best". When it comes to microbenchmarks, that's usually flawed. Very VERY few people actually get them right, benchmark.js is no exception.
That, of course, doesn't mean that benchmark.js can't be useful. For macrobenchmarks it will be roughly right. However, for something as small as leftpad, it's almost certainly not the right way to measure performance.
This isn't true - Benchmark.js will repeatedly rerun benchmarks until it gathers statistically meaningful results.
> There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark.
True, and there's actually nothing benchmark.js can do to ensure that doesn't happen in the general case but when this does happen the results are usually pretty obvious - we'd see billions of ops/sec. Incidentally the left-pad benchmarks do not suffer from this issue.
The article is completely consumed with getting the timing of benchmarking right. Which, to be fair, is a place where microbenchmarks often go wrong. It, however, isn't the ONLY place they go wrong.
This is the "benchmark lib" that was mentioned in the very first sentece of the comment you replied to.
https://github.com/left-pad/left-pad/blob/master/perf/O(n).j...
They are repeatedly appending a single character to the front of a string. This is actually O(n^2) so of course they are winning against it.
Anyway, it makes no practical difference in this case, since the one they labeled "O(n)" is the naive implementation that most people would write if they implemented left-pad themselves.
Quick googling shows that this optimization might exist but only for firefox and only if you use "unshift": https://lannonbr.com/blog/2020-01-27-shift-optimizations https://jandemooij.nl/blog/2017/12/06/some-spidermonkey-opti...
But it's very unlikely that the jit can optimize `str = ch + str;`
let padding = '';
for (let i = 0; i < len; ++i)
padding += ch;
return padding + str;
You’re a webdev aren’t you?
https://stackoverflow.com/questions/13865420/why-is-haskells...
I started with the problem of "I need a function that pads a string with a character out to some length". Coding it up took under 1 minute, easily under 5 minutes.
function leftPad (str, len, ch) {
const neededPadding = len - str.length;
if (neededPadding <= 0) {
return str;
}
return ch.repeat(neededPadding) + str;
}
The fact that the left-pad code has optimizations (which don't matter for the place I'm using it) and type-checks (which don't matter; my higher level unit tests would catch that mistake) is beside the point.
But I can maintain code that uses Entity Framework (or Dapper) much easier than I can maintain code based on a custom ORM that the “architect“ wrote.
I can count on one hand[0] the number of JS dependencies I've used in enterprise/large projects for extended periods of time where I've never needed to manually debug the library or read the source code. For enterprise software, sometimes the easiest solution is to directly hotpatch a vendored version of the dependency.
This is especially true with dependencies where bugs are fixed in major versions, but where upgrading and dealing with breaking changes would require significant code refactoring.
To drive the point home, I've been bitten by bugs in NPM itself.[1] Fixing that required reading through the source and manually swapping out one NPM's internal dependencies to a newer version.
And it doesn't matter if someone somewhere has had the same problem and posted it on the Internet unless I can find their answer online faster than I can fix the problem myself in my own library. Often this is not the case, filtering through issue trackers and trying to find the one blog post or comment that tells me how to solve the problem can be a big time sink.
[0]: Okay, maybe 2. But the point stands, it's not a rare or exceptional occurance.
Probably the worst was with a decently popular library someone had brought in, that tried to do a refactor from callbacks to async/await, without understanding at all how async/await worked. They'd leaked an async operation in the library code, so 'await'ing a specific function call in their API that returned a promise, didn't actually await everything the call was doing, ending up in a debugging nightmare. Of course their perfectly manicured suite of 8000 tests with 110% test coverage didn't catch it either, because the number of people who can write good quality tests is shockingly low, and library-writers aren't somehow magically ahead of the pack in that regard.
JS really feels like PHP did back when I was a newbie learning that shit. In other ecosystems, the 95th percentile devs seem to write all the libraries, so everyone comes here and posts repeatedly about how great dependencies are. In JS, it's the average dev writing all the libraries, and the average dev's code is enough to make my brain bleed.
I'm a big proponent of different advice for different ecosystems. If you're doing front-end JS, the pendulum has swung so far to one side that 'NIH syndrome' is treated like it's going to lead to the fourth reich, which makes 'chill out a bit on dependencies' pretty good advice if you're looking to get a leg up in the industry. But I'm sure there's other ecosystems where the same advice will just leave you with a tangled mess while your competitors leapfrog you in productivity with a good 3rd party dependency.
I'd say take any advice in threads like these with a grain of salt unless it's given in a bit of a narrower context. Taking some one liner about software engineering in general and applying it to your specific project is probably just a coin flip as to whether it's going to improve your code or not.
All of those properties make the code easier to reason about and test.
Being easier to reason about, and being a more exact abstraction for my needs, are both incredibly valuable properties.
I don’t have to care about how the underlying libraries work. I can treat them as a black box.
When I wrote haskell, the majority of the libraries did just work, and I didn't have to dig into their code to find bugs often.
When I wrote javascript, hundreds of the libraries I used did not just work. I usually had to care very much about their details because they were poorly implemented, full of bugs and incorrect abstractions, and often abandoned soon after.
I agree that there's benefits in reusing some well-socialized and well-implemented generic frameworks and abstractions. It's not worth using generic abstractions that are not well understood, buggy, and don't match your needs closely. In that case, write your own.
More generic is not always better. Above, I'm arguing that it's important for code to be easier to reason about. If a generic abstraction helps with that, cool, but it's not always going to be the case.
Any other scripting I do with Python. Any more complicated development it’s using a language with an ecosystem with adults - C# or Go.
Rob pike espoused "A little copying is better than a little dependency", and go refuses to add suitable abstractions to build generic reusable pieces.
Go has a strong culture of doing exactly the sort of thing I was talking about, and you were arguing against.
Almost everything I work with has bugs, so chances are I'm going to run into one. It's a lot easier for me to fix bugs when there are fewer layers and more of them are written by me. Of course, I can't write all the layers, but if they run on my service, I have to be prepared to fix them, or suffer from them being broken until a benevolent force fixes them for me. (Sometimes that happens, but usually not for the harder problems)
I've not profiled it, but I'm going to guess that now-a-days this will be faster than the current implementation on npm.
function leftPad (str, len, ch) {
str += '';
len = len - str.length;
if (len <= 0) return str;
if (!ch && ch !== 0) ch = ' ';
ch += '';
return ch.repeat(len) + str;
}
Static typing would actually fix this kind of coding error.
> Static typing would actually fix this kind of coding error.
That's not a coding error, that's addressing an oddity of JS coercion rules that a less experienced developer could easily have missed.
> if (!ch && ch !== 0) ch = ' '
That code says that if `ch` is falsey and not equal to 0, then set it to a space. The only arguable falsey value that should be excluded here is a literal `false`, but that's not a single character and is fairly ambiguous either way. I'd certainly fall on the side that a literal false should not be converted to `'false'` here.
> ch += '';
The next line converts to to a string by adding it to the empty string.
> return ch.repeat(len) + str;
So by the time it gets to this line we know ch is a string.
Static typing is great, but the bug you claim is there is not actually there.
It may specialize or inline, but that's a separate matter.
P.S. I think there's a bug here.
Step 1, inline repeat.
Step 2, remove the intermediate array allocation
Step 3, allocated a string array sized for the pad + str
Step 4, Use one of the many CPU instructions to repeatably copy the padding character and then the `str` into the same array of characters.
Nowadays there's a native padStart function and the left-pad package is deprecated as a result.
JITs only optimize hot code which means any benchmark without a warmup is going to measure cold + hot code time.
Further, JITs will optimize away unused results. They aren't using the leftpad results in any meaningful way.
What they are likely measuring is how long it takes for the JIT to optimize the benchmarking framework.
watch: https://vimeo.com/78900556 for how to microbenchmark a JITed language.
Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency[1]. The author of said library works on V8, and already considered all those problems and much, much, more[2].
[1]: https://github.com/left-pad/left-pad/blob/master/perf/perf.j...
Sure, It's popular. It's also wrong.
And yeah, I would and do write leftpad myself it it's not in the stdlib. But if there is a large library full of similar (string) functions that I might need, I would include that library. Not a singular dependency for this type of function.
Also I seem to remember that someone benchmarked the cached version and found it to be slower than the naive approach anyway. I could be mistaken there.
Numbers or ASCII-only-printing? OK that's a reasonable. Is there a desired overflow behavior?
Past that it becomes more an issue of where and why. The suddenly not-trivial example includes questions about fonts, layout, and multi-byte characters. Emoji, etc.
Incidentally, in pseudoscope:
Create a valid full-space pad string (termination / etc), then decrement back from the end of the source string and over-write the pad characters from the end to the start of the string, exiting either on no more pad characters or no more input.
A second algorithm might combine those two steps as one pass, fill the output buffer from back to front. Only for C style strings would this be an issue given the dynamic end point for the data structure.
This is why you time box things. Spend XX hours trying to get a thing working and if you aren't close, you grab a library and move on.
We all too often forget the scope: requirements, developing, testing, to say the least.
My favorite example is NPM. While the author has a point, I tend to rely on the wisdom of the crowd. Sometimes there is a reason why a couple of million developers - in the case of NPM packages - seem to be lazy.
In my experience, we ended up copy/pasting and modifying some code and syncing it with the "superfluous" package. Good intentions, badly executed.
Leftpad was the right itch at the right time and people found better ways to deal with NPM. NPM got better after that, as well as native implementations.
Better cope with NPM than fight it, my 2 cents.
Besides, in reality pulling in and using the dependency takes time as well. There's no real guarantee it's cheaper in terms of developer time.
Don't buy generalized statements like "programmers are always underestimate efforts needed", or even, for that matter, "a task always requires all the possible time it might take" (Parkinson's law). There are exceptions from them :) which sometimes, in a good team, look more than laws themselves.
Never do I give that raw 10-minute estimate to anybody, because it can be wrong by a factor of 10.
They just won’t have unit tests, and they’ll probably have lots of defects and other technical debt.