News for hackers. Stuff that matters.
Re:Malicious code in the purescript NPM installer by munificent on Monday July 29, 2019 @01:41AM (#20551778)
> It's just infinitely, obviously better.

Whenever a large number of skilled people do something for which an alternative is "infinitely, obviously better", there's a good chance that there is more going on than you know.

RubyGems used to be namespaced this way and moved away from it. They didn't do so lightly.

The problem is that ownership, and even names of owners change all the time. In the very very large majority of cases, this change of ownership is an implementation detail that doesn't need to impact package consumers. If you enshrine the owner's name in the package, it means any change of ownership is effectively a breaking change to the package. When you have very large transitive dependency graphs, the result is constant, pointless churn.

Re:Malicious code in the purescript NPM installer by hardwaresofton on Monday July 29, 2019 @03:10AM (#20552144)
It seems like a reasonable fix to this is to prevent name-specific owning (changing it to a group instead) -- this has the benefit that package maintainers who plan to maintain their packages forever can keep the same names, and those that don't can essentially fork their project and stop fixing the older version (@ <maintainer>/<project>) and force all changes to go to a new one (@ <group>/<project>) and hand off ownership as necessary.

This doesn't break old consumers and it allows for a pretty graceful migration path (if you want new updates, change your version) -- can eve be helped with marking package as deprecated or repo as archived and what not.

Re:Malicious code in the purescript NPM installer by akerl_ on Monday July 29, 2019 @03:34AM (#20552268)
As a thought experiment:

Assume Rubygems chose to operate in that way, where all gems must be owned by a group rather than an individual user.

Then, assume that the most flexible option is for each gem to be owned by a unique group: that way even if two gems are maintained by the same users right now, they use two distinct groups in case that ownership changes in the future.

We might as well just name the “group” the same as the gem name, since only one gem is managed by each group. So now the “purescript” group maintains “purescript”, the “pry” group maintains “pry”, etc.

As syntactic sugar for users, since the group name and gem name will always match, why make them type both? Let’s have all the commands support just referencing the gem name. If somebody wants to fork a gem and release it, their group and gem get a new name.

I think there’s a pretty compelling case that package managers should support group ACLing on publishing (giving multiple humans the first-class right to publish using individual creds to a group namespace, with the ability to add/remove users from the group over time. But once you’ve done that, the distinction between explicit group-name-in-package-path and changing-name-to-fork (so the difference between fork-group/orig-name and orig-name_fork-group) seems to shrink.

Re:Malicious code in the purescript NPM installer by thetrainfold on Monday July 29, 2019 @03:49AM (#20552353)
Could it help to force long names, preventing simple, 'elegant' names?
Re:Malicious code in the purescript NPM installer by akerl_ on Monday July 29, 2019 @03:55AM (#20552391)
For what it’s worth, I think I actually like the idea of mandating a 2-part namespace, as the way to force “long” names (where “long” means “with enough context to make forking easier and more obvious”). I just wanted to call out that “force namespacing” isn’t a silver bullet for the issue.

A better feature-add might be supporting dependency replacements. For example, pre-modules, Go had an issue where if you had a dependency and I forked it, I had to go through all my code and replace references to your import path with mine. If I depended on something that depended on you, I was out of luck (or had to vendor and regex or a variety of other hacks). Now, with go modules, I can do a “replace” in my go.mod and sub in my fork for yours. That has enabled me to be much more flexible with my use of forked repositories, and most languages don’t have a direct parallel.

Re:Malicious code in the purescript NPM installer by hardwaresofton on Monday July 29, 2019 @04:27AM (#20552557)
> Assume Rubygems chose to operate in that way, where all gems must be owned by a group rather than an individual user.

Sorry, this wasn't my premise -- I meant to have this as an option. As in <user>/<project> or <group>/project>.

> Then, assume that the most flexible option is for each gem to be owned by a unique group: that way even if two gems are maintained by the same users right now, they use two distinct groups in case that ownership changes in the future.

> We might as well just name the “group” the same as the gem name, since only one gem is managed by each group. So now the “purescript” group maintains “purescript”, the “pry” group maintains “pry”, etc.

I don't agree -- even if it's always group-a/purescript I think the distinction is still important, because in this case previous-group/puresecript still exists, but is frozen/archived. Here's how I'm understanding the scenario you laid out:

1. group-a/purescript is created

2. purescript changes ownership, group-b is going to be publishing it going forward

3. group-b/purescript is created

4. group-a/purescript freezes/archives/deprecates itself

5. group-b/purescript is actively developed

The fact that "group-b" is the "right" purescript is arbitrary/subjective to some degree.

> I think there’s a pretty compelling case that package managers should support group ACLing on publishing (giving multiple humans the first-class right to publish using individual creds to a group namespace, with the ability to add/remove users from the group over time. But once you’ve done that, the distinction between explicit group-name-in-package-path and changing-name-to-fork (so the difference between fork-group/orig-name and orig-name_fork-group) seems to shrink.

I think these two issues are a bit separate. Letting people dynamically change who owns/can publish to a repository is one way to solve this problem, but I think it's more complex than the fork-and-move approach.

IMO if some user wants to give up/transfer their repo, they:

1. find someone else to take over if they want

2. freeze/archive/whatever their repo

3. let the person fork & continue their work

An ownership change should be opt in, unless it was known @ package creation time that ownership would be a shared/rotated/changing/nebulous thing (which would be demonstrated by a group owning the package from the beginning).

Re:Malicious code in the purescript NPM installer by akerl_ on Monday July 29, 2019 @04:33AM (#20552585)
I wasn’t implying you claimed that all gems needed a group, I was proposing it as part of the thought experiment. My apologies if that was unclear.

To your list of examples: my point parallels your own, I think. I’m saying that given the “right” version is arbitrary and subjective, the difference between “group-b/purescript” and “purescript-group-b” is effectively nil. More concretely: if namespacing existed, you could fork “group-a/purescript” to “group-b/purescript”, but if namespacing didn’t, you could fork “purescript” to “purescript-group-b”. In either case, dependent projects need to update where they source their dependencies from.

Namespacing, in my experience, tends to make the forking process slightly “cleaner”, because you avoid having a potentially non-“right” “original” (for example, “purescript” tends to look more legitimate than “purescript-group-b”). But some comments in this thread seem to paint namespacing as a hard requirement, or claim that package managers without namespacing are missing a core, mandatory feature. The case I’m presenting is that this isn’t the case: namespacing is a useful feature for several workflows, but adding namespacing doesn’t fundamentally alter the issue.

Re:Malicious code in the purescript NPM installer by hardwaresofton on Monday July 29, 2019 @04:45AM (#20552646)
> I wasn’t implying you claimed that all gems needed a group, I was proposing it as part of the thought experiment. My apologies if that was unclear.

My apologies I certainly misread your comment.

> To your list of examples: my point parallels your own, I think. I’m saying that given the “right” version is arbitrary and subjective, the difference between “group-b/purescript” and “purescript-group-b” is effectively nil. More concretely: if namespacing existed, you could fork “group-a/purescript” to “group-b/purescript”, but if namespacing didn’t, you could fork “purescript” to “purescript-group-b”. In either case, dependent projects need to update where they source their dependencies from.

I agree -- the effects are definitely similar and almost equivalent. However does requiring a group/author change things at all? It seems like it could introduce an abstraction lever.

> Namespacing, in my experience, tends to make the forking process slightly “cleaner”, because you avoid having a potentially non-“right” “original” (for example, “purescript” tends to look more legitimate than “purescript-group-b”). But some comments in this thread seem to paint namespacing as a hard requirement, or claim that package managers without namespacing are missing a core, mandatory feature. The case I’m presenting is that this isn’t the case: namespacing is a useful feature for several workflows, but adding namespacing doesn’t fundamentally alter the issue.

I'm on the fence -- I'm not sure if this is a good counter case, but what about the layer of abstraction introduced by the implied/required existence of <group>? You could write code that imports "purescript", but then resolve it later (as some others mentioned, go.mod is or some other modules file that clarifies mappings) to determine which "purescript" that is. You could solve this by "alias"ing "project/purescript" to "purescript" (and then having some similar extra configuration that says "purescript" -> "project/purescript", and now I'm not sure if either is better (so basically having this indirection be a "module resolution feature" or a "module aliasing feature"), and if there's any value in forcing one (requiring the existence of <group> would almost certainly force the module resolution thing, but also break builds the second similarly named packages were published...) or if they really are just the same.

I also found the page on this by the rust team pretty convincing[0].

[0]: https://internals.rust-lang.org/t/crates-io-package-policies...

Re:Malicious code in the purescript NPM installer by akerl_ on Monday July 29, 2019 @04:50AM (#20552658)
For clarity, I think the comment referencing “go.mod” that you’re describing, at least in this thread, is from me :D

I think I agree that the core feature that impacts this issue is what go.mod solves, and what you’re describing: it should be easy and language-supported to sub in one fork of a dependency for another fork, so that users can flip between “group-a”’s purescript and “group-b”’s purescript, regardless of how the namespacing works on the module registry (notably, golang dispenses with a registry entirely: there’s no central system, except insofar as github is used for lots of people’s packages).

Re:Malicious code in the purescript NPM installer by munificent on Monday July 29, 2019 @01:26PM (#20557047)
OK, let's walk through how the scenario you lay out works in the context of a package graph.

I have my_app, which depends on foo and bar. Both of those use purescript. My app calls into foo which gets some object created from the purescript library and returns it. I then pass that object to bar. For this to work gracefully, they need to have a shared dependency on the same purescript.

(purescript is a weird example to use here, but imagine the shared dependency is a library that provides something like a reusable data structure.)

What happens to my_app when purescript gets passed from group-a to group-b? If foo wants to be on the latest, they need to move over to group-b. But if they do that and bar doesn't, then my_app can't get the latest version of foo. The foo and bar maintainers know that, which means they know they have a disincentive to move over to group-b. Better to stay on group-a and keep things moving smoothly with their existing users.

You end up in a situation where the choice that is better for a single package in isolation (move to the latest version of a dependency) is harmful to the package in context (it breaks shared dependencies and prevents users from upgrading to your latest version).

This is one of the most important situations to avoid when designing a package manager. As much as possible, you want to give package maintainers the freedom to evolve their package without it destabilizing the ecosystem. There's an argument that this is fundamentally what a package manager is — a tool to let you reuse changing code. If you don't need to evolve the code being reused, then FTP is a perfectly sufficient package manager.

Enshrining ownership in the package name directly confounds that. And, like another comment suggests, if you do that, maintainers will just route around it by creating an "organization" for each package, putting you right back where you started.

Re:Malicious code in the purescript NPM installer by sterlind on Monday July 29, 2019 @03:51AM (#20552363)
You could go farther and use a DNS name as a group name, then publish packages by signing with the SSL key. Anyone who doesn't want to shell out for a domain name could use a registry service that gives subdomains out. Why reinvent the governance wheel?
Re:Malicious code in the purescript NPM installer by hardwaresofton on Monday July 29, 2019 @04:31AM (#20552575)
I'd agree, but I don't think the additional complexity is worth it. I was prepared to suggest stuff like company.com/

That said, it's worked well for golang as far as I can see, despite 99% of the packages are github.com/<creator>/<package>.

Here are some writeups on what the rust team decided:

- https://github.com/rust-lang/crates.io/issues/58

- https://internals.rust-lang.org/t/crates-io-package-policies...

Their stance seems pretty reasonable though I'm not sure I would have done the same (and it's obviously very likely I would be wrong to do the opposite of what they did):

> Namespacing

> In the first month with crates.io 58, a number of people have asked us aboutthe possibility of introducing namespaced packages 90.

> While namespaced packages allow multiple authors to use a single, generic name, they add complexity to how packaged are referenced in Rust code and in human communication about packages. At first glance, they allow multiple authors to claim names like http, but that simply means that people will need to refer to those packages as wycats' http or reem's http, offering little benefit over package names like wycats-http or reem-http.

> When we looked at package ecosystems without namespacing, we found that people tended to go with more creative names (like nokogiri instead of “tenderlove’s libxml2”). These creative names tend to be short and memorable, in part because of the lack of any hierarchy. They make it easier to communicate concisely and unambiguously about packages. They create exciting brands. And we’ve seen the success of several 10,000+ package ecosystems like NPM and RubyGems whose communities are prospering within a single namespace.

> In short, we don’t think the Cargo ecosystem would be better off if Piston chose a name like bvssvni/game-engine (allowing other users to choose wycats/game-engine) instead of simply piston.

> Because namespaces are strictly more complicated in a number of ways,and because they can be added compatibly in the future should they become necessary, we’re going to stick with a single shared namespace.

Re:Malicious code in the purescript NPM installer by LoSboccacc on Monday July 29, 2019 @04:01AM (#20552428)
> this change of ownership is an implementation detail that doesn't need to impact package consumers

it does, because having a dependency means having a trusted path from the developer to the packager to the repository into your software and to your customers

frankly you'd want to do what you call "constant pointless churn" at every single version update, because god knows what comes trough your package manager, as this and many other articles pointed out repeatedly, and the fact that you are advocating to skip it not for version changes but for whole ownership changes is the opposite of a security oriented mindset.

Re:Malicious code in the purescript NPM installer by pjc50 on Monday July 29, 2019 @03:14AM (#20552165)
But every change of ownership is a potential mass security compromise.
Re:Malicious code in the purescript NPM installer by munificent on Monday July 29, 2019 @01:28PM (#20557064)
Every change period is. There's no guarantee that the same owner (individual or group) won't spontaneously go rogue and introduce harmful changes.

This is why modern package managers have lockfiles and give you control over when you upgrade any package version, regardless of ownership change. Ultimately, you are responsible for the code you reuse.

Re:Malicious code in the purescript NPM installer by gridlockd on Monday July 29, 2019 @09:22AM (#20554310)
> The problem is that ownership, and even names of owners change all the time.

If ownership changes, I want to know. It's perfectly fine to cause a little bit of breakage there that is easily fixed in a semi-automated, supervised way.

If a name changes, there could just be an alias, unless there's literally a trademark dispute underway, in which case the ownership-change process can be applied.

> When you have very large transitive dependency graphs, the result is constant, pointless churn.

A very large transitive dependency graph is a terrible thing to have. It shouldn't be made convenient to have it. Your package manager should scold you for it!

In all seriousness, if you have such a huge graph, then the amount of churn caused by package updates will likely be the dominating factor, not the change of ownership.

Re:Malicious code in the purescript NPM installer by munificent on Monday July 29, 2019 @01:33PM (#20557122)
> is easily fixed in a semi-automated, supervised way.

It's not. Once you have shared, transitive dependencies, the application author consuming a package is no longer the one who authored or is in control of the dependency on that transferred package.

> A very large transitive dependency graph is a terrible thing to have.

This is a fair subjective preference. Unfortunately, it flies in the face of reality. Anyone who maintains a package manager will tell you real-world package graphs are typically quite large and deep. If users didn't want that, they wouldn't do it.

> Your package manager should scold you for it!

Users don't generally like or use tools that scold them for doing what they want to do.

Re:Malicious code in the purescript NPM installer by mschuster91 on Monday July 29, 2019 @02:45AM (#20552038)
Why not resolve the owner name to some kind of uuid at time of package require/install and saving it in the lockfile? That way, a package upgrade can detect if the owner name has changed and fetch the correct package.
Re:Malicious code in the purescript NPM installer by gridlockd on Monday July 29, 2019 @09:31AM (#20554382)
This doesn't solve the issue where you type "npm i -s letf-pad" and your computer gets taken over by The Russians™.

Make people copy-paste the name, it decreases the chances of catastrophic failure.

Re:Malicious code in the purescript NPM installer by davnicwil on Monday July 29, 2019 @04:25AM (#20552545)
When package owners change, or change names, would the original package namespace still stick around for all the existing versions? Also, would there be a pointer built in that let you know about the new owner namespace when you try to upgrade automatically?

If so, I really don't see the practical problem. Actually it seems like useful information. If I'm upgrading and the package owner changed, that's definitely something I want to at least be told so I can look into whether that's important depending on the usecase. In that sense, it is a potentially breaking change. Particularly from a security standpoint.

If it's not the case, and everything including the version history is ported over to the new namespace and you're simply forced to change stuff just to get it working again, I agree this is just pointless churn.

Re:Malicious code in the purescript NPM installer by littlestymaar on Monday July 29, 2019 @06:24AM (#20553093)
> Also, would there be a pointer built in that let you know about the new owner namespace when you try to upgrade automatically?

And what would be the point of the namespace then ?

Re:Malicious code in the purescript NPM installer by davnicwil on Monday July 29, 2019 @06:41AM (#20553148)
> pkg upgrade @person-a/some-package

"You're on the latest version 1.2.3 of @package-a/some-package, however @person-a has officially transferred ownership of some-package to @person-b and there is a newer version 1.2.4 available at @person-b/some-package. If you'd like to upgrade, please update your dependency to @person-b/some-package"

Re:Malicious code in the purescript NPM installer by littlestymaar on Monday July 29, 2019 @07:19AM (#20553337)
That's the mechanism, I was asking about the benefit of namespace in that case. With this kind of ownership transfers you lose all the alleged security benefit of namespaces.
Re:Malicious code in the purescript NPM installer by davnicwil on Monday July 29, 2019 @08:40AM (#20553934)
I think we're asking the same thing, or I've not explained well. I'm saying when ownership transfers, it'd be good to keep all the existing version history on the original namespace, exactly so you're not forced to update all your dependencies in case you don't need to upgrade and/or don't want the updates of the new owner.
Re:Malicious code in the purescript NPM installer by _pmf_ on Monday July 29, 2019 @03:57AM (#20552402)
> Whenever a large number of skilled people do something for which an alternative is "infinitely, obviously better", there's a good chance that there is more going on than you know.

The Maven ecosystem has been doing things better than NPM for at least 10 years. The onus is on the NPM team for justifying their pathologically bad solution, it's also on the JS community to back off their "everybody can contribute" pipe dream. Maven works because in practice, we have some few organizations contributing packages, not thousands of individual, anonymous developers.

Re:Malicious code in the purescript NPM installer by gridlockd on Monday July 29, 2019 @09:36AM (#20554427)
I think the problem is that Java people are considered "old and lame", whereas the Javascript people are considered "young and immature", respectively. While this assessment is somewhat accurate, it inhibits the process of learning from each other.
Re:Malicious code in the purescript NPM installer by allover on Monday July 29, 2019 @04:10PM (#20558825)
> While this assessment is somewhat accurate

No it isn't, and please stop perpetuating this nonsense.

Re:Malicious code in the purescript NPM installer by _pmf_ on Monday July 29, 2019 @07:14AM (#20553298)
On further reading, it's debatable whether the "maliciousness" is not the community attempting to take away a project from its founder. Similar to how that toxic Twisted dev tried to push responsibility for maintaining their broken branch of Python 2 to the Python core lib dev.

You can fork, but then you're responsible for more than the "fun parts". The project founder might introduce measures to prevent fragmentation due to your bad decisions as part of damage control.

Re:Malicious code in the purescript NPM installer by twic on Monday July 29, 2019 @05:25AM (#20552807)
> a large number of skilled people

> RubyGems

Well, which is it?

Java has used namespaced packages since 1996, and it's been a roaring success. There have certainly been problems - for example, there is currently some absolute nonsense going on about the handover of big chunk of stuff from Oracle to the Eclipse Foundation [1] - but they have always been manageable, and don't come close to outweighing the benefits.

[1] https://www.infoq.com/news/2019/05/end-of-javax-package/

Re:Malicious code in the purescript NPM installer by littlestymaar on Monday July 29, 2019 @05:58AM (#20552979)
Java was indeed a success, but its dependency management is archaic at best. Yes it's better than FORTRAN or C, but I wouldn't use it as a reference in 2019…
Re:Malicious code in the purescript NPM installer by LoSboccacc on Monday July 29, 2019 @07:42AM (#20553464)
it's signed, hierarchical, decentralized, supports addressing multiple artifacts within the same module by type, includes a standardized way to retrieve sources or docs, supports pinning the version of transitive dependencies so you can manually resolve conflicts and on top of that can be extended to use completely alien package sources (which is unholy if you ask me, but it helped me survive the great osgi catastrophe of the 2010s thanks to tycho)
Re:Malicious code in the purescript NPM installer by kkapelon on Monday July 29, 2019 @06:34AM (#20553122)
what would you use instead as a reference?