However over 2018 we seem to have got to a point where I’m worrying about some basic professional competencies and communication systems inside the npm project. I don’t know much about it, but looking at it from a sideways glance, you have to start worrying about what it is that they’re getting wrong, and how to derail their mistakes in my own team and workflows.
and this is why writing yet another package manager instead of gradually improving existing ones - rpm, deb, etc. - is far from ideal.
The really confusing part is that the two are conflated by design: if building with Maven, your project's pom.xml also contains the declarative Maven configuration used to build your package, instead of containing just the package metadata (name, dependencies, etc). But that's not required, and it need not be part of the *.pom XML file deployed in the published package.
Since release, many build tools have been introduced that can consume and produce POM-compatible packages but don't require you to use Maven itself as your builder (Ivy for Ant, gradle for writing build scripts in Groovy, SBT for Scala, etc).
Anything counter to maven's way is a PITA. In this case, Maven dislikes platform specific binaries.
In the world of containers? Come on.
The conservation of complexity is at play here. Either you have a quasi-complex deb/rpm, and a simple Dockerfile with a RUN command installing it, or you have a quasi-complex Dockerfile which does all the packaging work itself.
That's exactly why I replied. deb/rpm in container is a reasonable choice, and that way, they aren't global for the system but local in the container.
For anybody who wants to build a package manager I highly recommend to take a look at what Paludis does (not necessarily how they do it).
The comparison isn't completely fair, as Paludis has a much better dependency resolution algorithm (in terms of quality), but I think the main difference comes from the package format. Ebuilds/exheres are Bash scripts which need to be evaluated in order to obtain basic information like for example build- or run-time dependencies. So to find out which packages should be updated, Paludis has to evaluate a few thousand bash scripts while other package formats allow much quicker algorithms.
Cave is just a newer front-end for Paludis which improves the user experience a bit, but the Performance should be pretty much the same.
This applies to FOSS in general, and then some...
One time out of fun and curiosity we flushed the cache and deleted the lockfile and the node modules dir, did npm i on a mac and a windows and they were undiffably different. Okay, this was months ago and probably there is a good explanation for it, but it was not reassuring.
Never had `npm i` break though... does it fail if you do `npm ci`? Are you pinning versions? Very peculiar behavior.
* NPM 6 stopped working with Node 4. Rather than actually fix it, they just left it broken for several days because it was already fixed in the upcoming release, and I guess they didn't want to do an emergency release: https://github.com/npm/npm/issues/20716
* There was a several hour period where you could publish packages, but then they would 404 when you tried to download the new version.
* They switched to using Cloudflare on Friday, and broke Yarn in the process: https://mobile.twitter.com/jamiebuilds/status/10001984632696...
* Somehow while switching to Cloudflare they blew away a bunch of the packages that had been published during the previously mentioned window. They also blew away all the versions of some packages. Last Friday night you couldn't 'npm install gulp'. Never gave any explanation for this: https://github.com/npm/npm/issues/20766
About a month ago there was a bug where npm would change permissions on a bunch of system files and render you entire system unusable in you ran it as root. But that was OK because the bug was on the "next" version of npm, not the one you'd get by installing 'npm install -g npm'... Except there was also a bug in the current version of npm that installed the next version of any package by default, so it did in fact blow up a bunch of machines.
Now, apparently, they are a teapot.
You simply have to endure it, and accept that every now and again, for reasons entirely beyond your control (and at a quite possibly very inconvenient moment) it's going to break.
This really annoys me, but what can one do? At least it's fixed now.
Install your own repository manager? This is the standard in every company I've worked for so far, at least in the Java world. Artifactory supports NPM, so set it up as a proxy.
This had to be added to a precommit hook to not break CI. Seriously, package.json should allow to specify what endpoints should be used if available in a given order. Now it's up to each dev team to handle.
My main concern is that it's brittle. Nexus caches exact versions and nothing more so we don't even have assurance that it will work nicely when NPM goes down.
On the other hand lockfiles are awesome. I missed them back in 2012... copying over node_modules on USB drives was not cool.
Do you have an externally available Nexus? Using ours through a VPN beats the main purpose - fast(er) installs. That's why for WFH scenarios we have a script to switch between our proxy and NPM.
If a developer is using an older buggy version of npm that doesn't respect .npmrc and changes a lock file to point back to npmjs.org entries, we deny the PR and ask for it to be fixed. Right now that check is unfortunately manual, but there are plans to automate it. It can be easy to miss at times though, since GitHub often collapses lock files on PR's due to their size.
For us, the main purpose of using Nexus as a proxy is to maintain availability and to cache/maintain package versions. If you're using Nexus to make things faster, then you probably shouldn't be using it. If you want faster installs, look into using `npm ci`.
To ensure that you're actually deriving benefit from your Nexus install, you have to block outbound connections to the NPM public registry from your CI build agents (if you don't firewall it off, you don't want to wake up one day and find that both origin and the proxy are erroring because your proxy never actually cached anything and you never tested tested your proxy... right?), with only the Nexus installation permitted to make such outbound connections. And as bad as NPM may be, there are real maintenance costs to running your own Nexus install (not least of which, managing updates that will take Nexus down and communicating them with your dev team so that CI builds which error out when Nexus is down can be restarted when it goes back up), and thinking that you can do better than NPM is hubris. Running a private Nexus OSS install for the purpose of trying to increase availability for low cost (not zero - you still have to pay the infrastructure costs) is usually a false economy.
If you work for a company with enough operations and infrastructure resources that adding a clustered install is trivial, then you probably have enough resources to pay for an Artifactory license.
TL:DR - NPM has its faults but it's still probably de-facto both more available and better updated than taking on the responsibility of running a proxy unless you have mature ops/infra teams
https://github.com/rlidwka/sinopia/issues/456
Verdaccio, mentioned in a sibling comment seems to be the recommended replacement ...
This problem is not limited to npm. I remember a few years back there were similar issues with RubyGems, where it'd go down leaving many developers unable to deploy.
Heck, how many projects do you think would be left unable to deploy if GitHub went down? I remember a few years back they'd have their occasional issues and Twitter would become a storm of angry developers.
For many projects not being able to deploy or develop at any moment, as well as dealing with left-pad style issues is implicitly accepted as a reasonable trade-off.
The weird thing is, open source development is as close to a free market as one can get. Frustration with NPM should result in multiple javascript package managers competing to undermine NPM's dominance, but the only alternative is one that uses the Node registry.
Maybe we'll go down the clone route if the cost of npm issues becomes too high relative to the hassle and expense of maintaining our own.
The status quo in which anything that goes wrong breaks the entire universe because no one vendors is also not ideal.
However, consider the long list of disasters that have occurred with NPM recently, and how many of them would have been less disastrous had vendoring been the exception rather than the rule. Left-pad wouldn't even have been an issue, for example - builds simply would have been unable to update, but nothing live would have been affected.
To misquote Ben Franklin here, the tradeoff isn't between a greater ideal and a lesser ideal, but between security and convenience.
I wrote a post a while back about using Yarn's offline mirror: http://blog.isquaredsoftware.com/2017/07/practical-redux-par...
If you are updating components along with business rule updates, then, yes, it’s going to complicate code review, regardless of vendoring.
Explain yourself better or top spreading this nonsense. What do they give you that you couldn't develop without them?
Everything else is legit though.
nomnom is no longer maintained, so npm takes the project over, & makes a 2.0.0 update which is empty besides a README saying it's deprecated. Which breaks some packages which have a '>=1.2' version. So npm deletes 2.0.0. Which breaks for people who have had their package.json update from 2.0.0. Even when they'd had a system working with 2.0.0 (not even using nomnom, just a depedency of a dependency for the commandline portion that isn't being used) & were ready to deploy
In Rustland you can't publish a crate with '>=1.2' dependencies
That was the end of February, so not sure but that may be it.
Imagine uBlock Origin's Chrome extension author creds were hacked. "He" publishes a new version of the Chrome extension that monitors coinbase.com and fakes the transfer/confirmation screen, or submits transfers in the background. The extension has "write" access on all sites, so the rogue extension can also monitor your Gmail and silently inject a filter that routes trade confirmations to trash.
Or the "requests" library in Python gets an update to replicate 2FA codes via Twilio to a 3rd party.
Sure, you can do pinning and cryptographic signatures to verify that v 1.0.0 of X is really what you expected.
But who audited 1.0.0 of X in the first place...?
We are one step away from very bad shit hitting the fan in a very painful way... so let's pretend everything's fine and try not to think about these things.
https://www.reddit.com/r/programming/comments/886zji/why_has...
So in case you weren't aware: If you're using webpack, you're using is-odd. Which itself relies on the excellent package is-number.
So, how and why did the developers let things get this bad?
If you care about the size of the code you are delivering then it makes sense to skip some larger math/numerics library and instead opt for is-odd, which itself includes is-number.
Unfortunately, when the authors of modules also do this and not just end users, it becomes hard to track down how much a dependency actually adds, and the chain of dependencies can get very deep. Once that chain is deep we lose the ability to easily determine how a change the complex system we now rely on.
And that's all really at a surface level. Once we start talking about trust and security, the problem gets much worse.
"My package has 2 million downloads, I want x% more than market rate" attitude.
It's like the equivalent of influencer marketing, but for code packages. Of course I could just be a crazy old man. Who knows?
They specifically target noob questions/discussions, post links to their blogs that talk about certain dependencies. The articles themselves have very little substance, they're full of buzzwords and spend all their words making ambiguous claims, for example they'll say that the dependency makes your code "scale", but won't give any examples from a large code-base, especially not any comparing to the alt case (without the dependency). They make claims about performance, but don't elaborate, don't show benchmarks (or the'll show benchmarks that don't apply to 99% of projects, kind of like how immutableJS is obviously not faster unless you're working with big data structures, which you almost never are in your standard CRUD app where everything is paginated server side).
This shit is everywhere, and there's a lot of people with a vested interest in making sure these tools propagate as much as possible.
But no matter the joke, in no other language you 'd see packages like "is-true", "is-false", "ifAnd" (because && is too difficult /s) and "ifOr" (because... what's that "ll".... wait, is it "II" ? Um... How do I type it that vertical || line thing again?? /s).
The problem is not with users creating "useless" packages, the problem is with the community embracing them and having completely redundant one-liners gather million of downloads every week.
It just feels... icky. As if the whole's thing is put together with duct tape and bubble gum. We didn't resort to these kind of things when we had to support IE5.5 and 6. What went wrong from 2010 till today?
Seeing "is-true" rely to "is-object"... Why? Seriously, why? Maybe I am being completely narrow-minded and bitter.
If that is so please teach me and I 'll try to be better. I am not trying to provoke only to learn.
I think it's easier to answer in reverse order. DISCLAIMER: This is part rant, part "projected-constructed worldview". I absolutely welcome balloon-popping and "that's absolutely not true" and pointers to things that debunk/disprove points, so I can have a proper idea of history, and a better personal idea of the way things are.
First of all: the Web is "new media". Yep, fussy annoying buzzword; but one that has some truth to be discovered in it.
The position/power/operations/general relevancy of companies like Google, Mozilla, Facebook, Amazon, etc, more closely resembles that of CNN, Fox, CBS, Disney, etc around the 1995-2002 timeframe, than AT&T, AOL, CompuServe, Verizon, Comcast, etc, in the same time period. These modern companies have taken up the slack of the old TV/cable media empires of recent yesterdecades.
The Web has taken the place of TV, not by replacing it, but by becoming the new media focus.
It's a bit of a good side-illustration to mention that the Web is the perfect example of something with "Shiny Actually-New Thing Which Seriously Truly Isn't Anything Like What Came Before It" syndrome, for exactly the reason I mentioned in italics - the Web, for most people, resembles nothing like TV.
I think one reason for this is the technological generation gap: older people don't fluidly relate to computers, while the younger generation don't have quite the same perception/appreciation of non-interactive TV that their elders do, so while there is some collective conversation about media agendas, it doesn't bridge the generational divide that coincidentally happened with TV>Internet, and so people don't realize the _exact same_ control agendas that went into TV now go into the Web.
But technical folks who've seen the Web grow, and change, wonder: what's happening? My working theory is that the technical changes we're seeing stem from various upstream political machinations within media.
So, to the 2nd half/2nd question. As a "new media" company, Google wants to retain captive control of as much as possible, in exactly the same way $oldmedia would. So what do they do?
Well, back in the 80s and 90s you'd run cable to the entire country to control distribution, cultivate deals and negotiations with advertisers and producers, etc etc. See Also: CNN/MSN/MSNBC/CNBC (I never did figure out what acronyms went where with those companies), Comcast (which I understand doesn't just provide garbage cable/DSL internet, but is also a TV and media network), Time Warner, etc.
So, nowadays, Google's pretty much nailed the advertising thing, undoubtedly to the chagrin of everybody else. On top of that they handle the world's searches - and you never lie to a search engine (!), and that's the world's biggest non-perceived captive audience ever right there.
But they're GOOG; they legally have to vacuum up as much of everything that everyone's perception of them supposes is within their purview, in order for the balance sheets to look right. That most definitely includes Web standards. So, how do you control Web standards?
Well, you apparently approach your cofounder and say you need to build a browser.
I wonder what the friction breakdown was - how much political, how much technical - when Google split with Apple working on WebKit, and forked/created Blink. Heh, I'll never know. But the commit rate sadly plummeted after the fact (http://mo.github.io/2015/11/04/browser-engines-active-develo...) so code quality certainly wasn't on the decision table; something else was the priority. My guess is that it was straightforward market dominance, and competing against WebKit (and iOS!).
So, (having won) one Internet (marketshare) later, now what? Well, now you have a browser, you can play in the "democratic browser vendor" sandpit as a first-class citizen! The Standards® Committees™ are full of people who absolutely love discussing Web minutae all day; all you have to do is fund conferences and shows for them to attend, and they'll go ahead and build their own candles to fly around. They're like the self-organizing equivalent of https://xkcd.com/356/, it's wonderful.
They do need feeding though; specifically, you need to feed them just slightly too much in terms of upstream changes, so they can almost keep up but not quite. Just on the edge of being able to manage everything though. This is easy to do by, instead of adjusting the workload slider, adjusting the paid employees slider instead; and you can indirectly do this by doing nothing to democratise/vindicate/humanize the standards managlement process at the W3/WHATWG level so it seems obscure and academic and boring; and then it won't seem intuitive for upper management to prioritize funding for it, and voilà.
Now you just need to find a good source of chaos to keep everybody on their toes. You do this by investing heavily in coming up with new ideas for the standards committees to implement, keeping careful track of what everyone's discussing, and prioritizing what to carefully drop quietly on the floor ("oh yes we do need to get back to that sometime sorry about that") and what to react to as an interested browser vendor. Be polite enough about other vendors' technical interests, of course; your market share will typically dictate what'll actually get maintained at the end of the day. Pick things to cooperate on to keep everything looking democratic, so you don't get cornered into implementing things you don't want to (if this even happens?).
Oh, regarding the democratic bent that seems to underpin everything related to the Web (and open source for that matter): I consider it all a psyop (I was recently reminded of the existence of this sort of thing by https://news.ycombinator.com/item?id=16918752). Focusing on social organization (and justice and related fields, FWIW) are excellent ways to sink/drain massive amounts of energy; get humans into that loop and they don't seem to be able to disentagle themselves thanks to peer pressure and the precedence that gets created. (This is why "minority groups", typically the gender-focused ones, are in the permanent media spotlight - it allows control.) Democracy is a foundational part of the US Constitution, so of course democratic leadership is a very politically correct way to do "high-level" "important" things like directing the future of the Web; on top of that, democratic "leadership" is the kind of thing most people see straight through unless someone pokes the hologram the right way (especially when everyone's preoccupied with getting their social dance moves right), so I can easily see it being used as a tool in situations like this.
In the case of Web standards groups it was particularly easy to deploy "democratic leadership" - the Web groups seem to have been engaged in a most fascinating set of caricatured circling/courting "everyone's voice is important" dances since time immemorial. I was enlightened about the historical state of the Web a couple years ago: https://news.ycombinator.com/item?id=10684426 (I accidentally locked "i336_" by setting "showprocrast" too high, woops). Besides the antitrust bits I commented on, I've also since learned that IE6 implemented an early draft version of the DOM (or it could have been the CSSOM) before it got redesigned from scratch, then the necessary reimplementation work was never approved, and this was the source of all the rendering bugs. (I need to go re-find that link, sorry!)
Reading through those mailinglist posts showed me that the various Web working groups have always been... I'll say it, somewhat stuffy, and this fact is not a new development. It's sadly so easy to psyop-weaponize though (or, in any case, the baseline likelihood that this has happened is so incredibly far away from absolute zero...), as per the xkcd I mentioned before. Meanwhile, you carefully build EME behind the scenes (the implementation of which is kinda depressing; see zb3's reply to https://news.ycombinator.com/item?id=15796420).
Unfortunately some people run off and actually go and make interesting and amazing things in the face of great odds, for example https://codepen.io/AmeliaBR/post/me-and-svg (<> https://news.ycombinator.com/item?id=14155393). It seems those who stubbornly want to succeed are handled by ignoring them and hoping they'll go away. Or - https://en.wikipedia.org/wiki/Hanlon%27s_razor - maybe it's that whoever it is is calling the shots is so laser-focused all they have is perfect indifference since these cool things not within the scope of their media/other agenda. An intersection of explanations is valid. But the features these amazing people come up of course get used in any case, IMO without due acknowledgement too.
Most Web developers seem to get caught up in the entropy and chaos of continual change and development, get blind{sid}ed by it, and don't really [get the chance to] form a foundation strong enough to get them out of the slipstream, looking far afield, and doing something different and unique. Commercialism and existential "must have a job to eat" probably has a big part to play in folding in to this, along with mental health impacting society's ability to think clearly, along with all the other issues the various in-vogue generations apparently focus on.
To do one of those zoom-from-outer-space-into-a-tiny-backyard things and zoom/focus from this wide view into NPM packaging, I actually argue that the state of the JS ecosystem is easily explainable by saying that the Web doesn't have a standard way to do libraries, so everyone solves that problem by vendoring (including/bundling) everything they use. This solves everything for Node.js and the browser.
> Continued >
Fundamentally, on the ground, you're looking at economy of scale, both in terms of implementational scale and in terms of humanity collectively responding to a problem and scale and slowly annealing (if I can borrow that from "simulated annealing") toward a solution that just happens to work. The macro level works out; the micro level looks terrible. Such is how this kind of thing works out; social scale/crowd dynamics are _really_ fascinating to watch at work.
Node.JS happened, so suddenly JavaScript needs to run on servers and in browsers, and handle backend (Databases! Legacy stuff! FoxPro! How To Port Your Visual Basic Excel Macro To JavaScript! I just found two libraries for RS485!), along with frontend (Win32 event loop inspired Rube Goldberg machines!), and everything in between (......uhhhh...... all of GitHub?). The ecosystem and infrastructure - all the library code - has to be compatible with everything and everything else, across more domains than I even realized existed until I thought this through 5 minutes ago. The easiest way to work in the browser and also work on the backend apparently is to fragment everything out, because nobody's come up with a better way to stdlib JavaScript. The jQuery-esque approach of "this system works unbelievably well within its problem domain but doesn't generalize" effectively died when jQuery aged beyond its point of relevance - and because the plugin system was so locked-in to jQuery, all the plugins died with it. That type of approach is very very top-heavy and simply didn't scale. It made sense when all we had was browser [delivery] though, backend types hadn't gotten their mitts over everything and made compilation pipelines for everything, etc. Now that's happened, it makes more sense to `cat * > out.js` (well, in more LOC, anyway) because it just scales better.
JS feels clunky, but once you look all the way down, I feel like the vertigo actually causes things to kinda make a tiny bit more sense and fit into place a bit better. Maybe.
I find it not unreasonable to use the explanations above to justify the higher-level "why???" of the way things are.
The duct tape and bubble gum isn't JS-specific. It's endemic to all of technology. Like with media, there are vested interests in things working out certain ways with security and whatnot. (My current favorite meme of this is http://web.archive.org/web/20160409181051/http://article.gma..., and I also found https://news.ycombinator.com/item?id=17160109 the other day and that was kind of very fun to read, and I also read the other day that the NSA discovered Heartbleed "very shortly" after it "appeared" in 2012 or whenever it was, which was a TIL of its own)
And so it is that we use systems that operate at cross-purposes with how we think and incorporate high-friction impedance mismatches, and which raise the chances we'll make mistakes, which affect everyone: https://twitter.com/x0rz/status/865196993215442944, https://sites.google.com/site/testsitehacking/
The above being said, I do completely understand your frustration with using Node. I have the same gripes with PHP, which I learned ages ago and Really Really need to move off of, but it just runs so incredibly well on older hardware and has zero warmup time... like 0.2ms on my 12 year old laptop... sigh. One day I'll find something, or maybe I'll make it myself.
I'll leave you with three last links which I couldn't fit into the narrative, but which I think are relevant/related.
Firstly this an interesting article that helped me properly understand Google: https://medium.com/p/e836451a959e
Next, I wanted to call out this NNTP post from 1995 (linked from "sent out via" in the above) showing storage investments: https://groups.google.com/forum/#!topic/mail.cypherpunks/4CD...
For context about cost in the above link: I found this very cute little document while randomly manual trawling (I love finding things not in Google's index) whose HTTP headers say it's from 1996: http://users.monash.edu.au/%7e%6a%6f%68%6e%6d/%77%65%62%73%6... (#1 I don't want its filename to be indexed just yet :P and #2 I want to do an experiment to see if the crawler resolves the escapes and finds the URL anyway)
NB. If you have a better term than "new media", I'm curious. It's such an easy term to overload. The symbol picked for biohazard was designed from scratch to be meaningless so a meaning could be cultivated for/associated with it over time, and that cultivated meaning would be what "stuck". (I wonder how many other things this has happened with.) https://en.wikipedia.org/wiki/Hazard_symbol#Biohazard_symbol
NB2. If you - say - accidentally hit F5 while writing a post... and you're on Linux, suddenly get really really interested in opening Chrome's task manager, identifying the renderer PID and SIGSTOPping it in as few seconds as you can. The first ~15% of the above text was recovered via gdb generate-core-file \o/
NB2a: (For completeness, if you accidentally close the tab, have working hibernation set up, and either go fishing in the swap partition on resume, or if you don't want to risk stepping on anything in swap on reboot, carefully boot something that won't automatically swapon.)
NB3: $lists_i_am_on += 5000 :P