News for hackers. Stuff that matters.
Re:Malicious software libraries found in PyPI posing as well... by Sir_Cmpwn on Friday September 15, 2017 @08:07AM (#15256409)
I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human being verifying the upstream and keeping malware out, and get more consistency across packages as a bonus. If you want your package added it's as simple as sending an email and provides a new avenue for people to contribute to the success of the ecosystem as package maintainers.

When you make the next big thing, consider this approach.

Re:Malicious software libraries found in PyPI posing as well... by drdaeman on Friday September 15, 2017 @08:28AM (#15256528)
Maybe you're right, but I see one possible downside that is quite important.

I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice.

With the intermediate maintainers, that's going to get worse.

I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) - a little like it's on the Docker Hub, where curated mainstream images are just "debian" or "python" but anyone can upload e.g. "jdoe/debian" if they need some customization.

Re:Malicious software libraries found in PyPI posing as well... by nyrikki on Friday September 15, 2017 @11:53AM (#15258139)
PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file.

I do wish that `--user` was the default for pip.

It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839155

Although Debian/Ubuntu default to --user on pip people resort to sudo because the current standard user bin directory isn't in the default path due to a regression.

I may start a project to create a apparmor/selinux wrapper for pip to audit and restrict access to sensitive resources. I actually have a fairly heavyweight version in place on my build pipeline to detect new dependencies. I add the files/network resources that a build accesses outside of the testing stage to the build artifacts. But it wouldn't be cross platform enough for Windows/Mac.

Re:Malicious software libraries found in PyPI posing as well... by drdaeman on Friday September 15, 2017 @06:13PM (#15261220)
1. This would require PyPI to provide computing resources to build the packages (for all OSes, if the package contains native code). And then, malicious packages would just detect the build environment and avoid exhibiting the unwanted behavior. I think I read that it's not unusual for malware to detect being ran in a VM and doesn't do anything suspicious to complicate detection and analysis.

2. I don't think `pip install --user` adds any significant security. A little bit - sure, but not much. A trivial injection into ~/.*shrc or ~/.profile (I don't think anyone would notice the file was changed until it's too late) would result in full system compromise on the next login and sudo invocation. Same goes if you have ~/bin or ~/.local/bin (or anything user-writeable) in $PATH.

And even with non-root access, malicious software can do a lot of undesirable things (e.g. send spam or steal user data).

---

I believe, signature-based trust (with mandatory code signing) is the way to go. On the first `pip install` ever ask "The package not-expect (1.2.3) is signed by The Spanish Inquisition (key: ...) and was audited by The Python Developers (key: ...). Have you verified the keys and do you trust a) this vendor, b) this auditor or c) both?", then it gets recorded in ~/.pip (for this machine) and ./requirements.txt or ./setup.{py,cfg} (for distribution) so future installations don't ask anything.

To get non-interactive mode (for CI or something), on must either pass something like --insecure-skip-signature-checks (so they mean it) or pre-supply all the trusted keys.

(Not ideal, of course - just a quick idea. Surely, it has a lot of rough edges to polish.)

Re:Malicious software libraries found in PyPI posing as well... by nyrikki on Friday September 15, 2017 @09:56PM (#15262358)
On point 1) yes so they would have to compile, that is not a huge barrier and it would also result in improved quality of modules just as CI/CD does.

On 2) even if you ignore the much larger attack surface due to running ALL installs as the root user, consider the one-shot opportunities by disabling the protections of capabilities(7) # capsh --print -- -c 'pip list > /dev/null' Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep

And if --user was the norm, it would be trivial to write an apparmor/selinux policy to protect files like ~/.*shrc or ~/.profile

It is just the basic principle of least privilege. Heck if irssi can bother with an apparmor profile, the maintainers of pip or the package should be able to.

Re:Malicious software libraries found in PyPI posing as well... by rplnt on Saturday September 16, 2017 @08:29AM (#15264347)
With this system, one could pull a Volkswagen.
Re:Malicious software libraries found in PyPI posing as well... by zzzcpan on Friday September 15, 2017 @08:48AM (#15256651)
I don't think there are any maintainers that verify upstream code, they only manage packages and updates. Which is actually safer to do without maintainers, completely automatically, as it will eliminate a huge attack surface introduced by a maintainer.
Re:Malicious software libraries found in PyPI posing as well... by ex_amazon_sde on Friday September 15, 2017 @08:55AM (#15256701)
Debian Developers are responsible of the package quality. Most package only well-known software that they are familiar with or read the code. Some do more thorough security audits.

Some security-sensitive packages are maintained by teams to share the workload.

Re:Malicious software libraries found in PyPI posing as well... by zzzcpan on Friday September 15, 2017 @09:18AM (#15256870)
They do a lot that could be and sometimes even is automated, but that doesn't help with security, only weakens it. Which is my point. It's better to automate package generation and trust fewer people, mainly the authors, not introduce maintainers into it.
Re:Malicious software libraries found in PyPI posing as well... by peterwwillis on Friday September 15, 2017 @09:53AM (#15257138)
Authors don't generate distro packages because there's too many distros and each distro needs to make changes that have nothing to do with code. Maintainers are necessary and a package signed by the author is a non-starter.

However, source code can be signed and then used to make a package signed by a distro.

Re:Malicious software libraries found in PyPI posing as well... by zzzcpan on Friday September 15, 2017 @10:21AM (#15257360)
Think of it instead of allowing access to the system to maintain packages, we allow people to submit code that generates packages.
Re:Malicious software libraries found in PyPI posing as well... by peterwwillis on Friday September 15, 2017 @11:06AM (#15257725)
Of course you can do this - all packages are basically just wrappers around upstream code. But you still need someone to maintain the wrapper, and they have to check every new code release to see if there's something in the wrapper that has to change. And there are multiple distros. There's no getting away from maintainers with traditional linux distros.

Code package management is different. The author writes their software specifically to conform to the one code package management system. There's no wrapper glue needed, so you don't need a maintainer. Just release your new code and it fits into the system, and other code/tools/etc can just pick it up and use it.

This works if you constantly update all the software you use everywhere, and is pretty much guaranteed to become a nightmare if you don't. CPAN is probably the most mature software package management system in existence and it's still a nightmare if you don't keep a private repo and tightly manage releases, and you absolutely need a maintainer.

Re:Malicious software libraries found in PyPI posing as well... by zzzcpan on Friday September 15, 2017 @11:32AM (#15257949)
To be clear, what I'm suggesting is to generate those wrappers automatically, instead of maintaining them manually. A script can visit a release page daily, parse it and check for updates. If there is a new upstream release, it can generate a wrapper and let the build system do the rest, produce binaries, test them, etc. When things break, the code needs to be fixed, but it's definitely very far from every release. And you don't have to trust the maintainer of that script anymore or even have a separate maintainer, everything could be reviewed on pull requests with only a small group of people having commit rights to the repository.
Re:Malicious software libraries found in PyPI posing as well... by peterwwillis on Friday September 15, 2017 @02:33PM (#15259512)
That's basically how packages are maintained today, they just don't have as much automation and there's a lot less packages as a result. If you made one package for every software update and had to review each one you'd spend a lot more time reviewing.

Trust isn't an issue in reviewed/maintained repos because you have eyeballs on everything. When anyone can just ship an app/library and release it automatically you get these malicious software issues.

Re:Malicious software libraries found in PyPI posing as well... by kasabali on Friday September 15, 2017 @05:44PM (#15261029)
> I don't think there are any maintainers that verify upstream code, they only manage packages and updates.

Even that minimal amount of work is enough to prevent such a ridiculous attack like typo squatting.