News for hackers. Stuff that matters.
Re:Malicious PyPI packages stealing credit cards and injecti... by ericpauley on Tuesday August 03, 2021 @09:26AM (#28048859)
Out of curiosity, is it really necessary to have the separate artifact server? Pinning dependencies by hash ought to be sufficient.
Re:Malicious PyPI packages stealing credit cards and injecti... by yardstick on Tuesday August 03, 2021 @09:32AM (#28048932)
Seems reasonable enough depending on your use case.

In our situation we store our own (private, commercial) artifacts as well as third party ones, so we already need to have a server, and we know our server is configured, maintained & monitored in a secure fashion whereas I have no guarantees with public servers.

Plus our build servers don’t have access out to the internet either, for security. Supply chain attacks like SolarWinds and Kaseya are too common these days.

Edit: Also, our local servers are faster at serving requests, allowing for faster builds, and ensures no issues with broken builds if a public repo went offline or was under attack.

Re:Malicious PyPI packages stealing credit cards and injecti... by jrockway on Tuesday August 03, 2021 @09:52AM (#28049170)
Security and availability don't have to be mutually exclusive. I remember in the early day of Go modules our Docker builds (that did "go mod download") would be rate-limited by Github, so a local cache was necessary to get builds to succeed 100% of the time. (Yes, you can plumb through some authentication material to avoid this, but Github is slow even when they're not rate limiting you!) Honestly, that thing was faster than the official Go Module Proxy so I kept it around longer than necessary and the results were good.

Even if you cache modules on your own infrastructure, you should still validate the checksums to prevent insiders from tampering with the cached modules.

I'll also mention that any speed increases really depend on your CI provider. I self-hosted Jenkins on a Rather Large AWS machine, and had the proxy nearby, so it was fast. But if you use a CI provider, they tend to be severely network-limited at the node level and so even if you have a cache nearby, you are still going to download at dialup speeds. (I use CircleCI now and at one point cached our node_modules. It was as slow as just getting them from NPM. And like, really slow... often 2 minutes to download, whereas on my consumer Internet connection it's on the order of 10 seconds. Shared resources... always a disaster.)

Re:Malicious PyPI packages stealing credit cards and injecti... by helsinki on Tuesday August 03, 2021 @09:56AM (#28049212)
Tangential non-sequitor:

> self-hosted on AWS

People forgot what self-hosting actually means.

Re:Malicious PyPI packages stealing credit cards and injecti... by jrockway on Tuesday August 03, 2021 @10:04AM (#28049314)
I disagree with that analysis. That basically means running some binary yourself, with the alternative being buying a SaaS product that is hosted by the developer. You may think that "self-hosting" means having your own physical server, but I have never heard anyone else use the expression like that.
Re:Malicious PyPI packages stealing credit cards and injecti... by paulryanrogers on Tuesday August 03, 2021 @10:06AM (#28049329)
Meanings can evolve over time. I tend to think of self hosted as installing from code and managing myself, whether on local hardware or remote.
Re:Malicious PyPI packages stealing credit cards and injecti... by jijji on Tuesday August 03, 2021 @10:29AM (#28049625)
The OP wrote "self-hosted Jenkins", which has a different meaning
Re:Malicious PyPI packages stealing credit cards and injecti... by zerkten on Tuesday August 03, 2021 @09:40AM (#28049012)
It's going to depend on your circumstances. You don't share any context about your app or any of your development. Rather than looking at this from the perspective of needing an artifact server, you just look at it as a case of supply chain protection.

If pinning dependencies counters all the threats in your threat model, then fine. If not, you need to be doing something to counter them. An artifact server, or vendoring your dependencies, provides provides a lot of additional control where chokepoints or additional audits can be inserted.

If there was no management cost or hassle then you'd just have an artifact server to give you a free abstraction, but it's a trade-off for many people. It's also not a solution in itself, you need to be able to do audits and use the artifact server to your advantage.

The problem is really with the threat models and whether someone really knows what they need to defend against. I find that many engineers are naïve to the threats since they've never personally had exposure to an environment where these are made visible and countered. At other times, engineers are aware, but it's a problem of influencing management.

Re:Malicious PyPI packages stealing credit cards and injecti... by fortran77 on Tuesday August 03, 2021 @09:31AM (#28048912)
It's nice to have a build machine that can complete a build when it's disconnected from the Internet
Re:Malicious PyPI packages stealing credit cards and injecti... by kawsper on Tuesday August 03, 2021 @09:32AM (#28048929)
How often does that happen?
Re:Malicious PyPI packages stealing credit cards and injecti... by oauea on Tuesday August 03, 2021 @09:42AM (#28049043)
Re:Malicious PyPI packages stealing credit cards and injecti... by jrockway on Tuesday August 03, 2021 @09:56AM (#28049214)
Docker Hub also has rate limits and outages, so yet another thing you want to cache if you promise customers "we'll install our software in your Kubernetes cluster in 15 minutes 99.95% of the time".
Re:Malicious PyPI packages stealing credit cards and injecti... by kawsper on Wednesday August 04, 2021 @07:52AM (#28059891)
I don't see how any of those are build machines that are offline.
Re:Malicious PyPI packages stealing credit cards and injecti... by oauea on Wednesday August 04, 2021 @09:50AM (#28060972)
Build machines generally want to be 'online' so they can connect to the internet. They want to connect to the internet so they can access services hosted on the internet. Many of those services use Cloudflare, Akamari or Github. If any of those are down, those services are down. Now it's great that your build machine is online, but the packages it's trying to fetch won't be available because their CDN is offline.

Now if only you had a local caching proxy.

Re:Malicious PyPI packages stealing credit cards and injecti... by 83457 on Tuesday August 03, 2021 @09:40AM (#28049016)
Sometimes I use an air gapped test lab. Setup of certain software and projects is a real pain. Sounds like this approach could help.
Re:Malicious PyPI packages stealing credit cards and injecti... by ex_amazon_sde on Tuesday August 03, 2021 @11:24AM (#28050278)
It's standard practice in many large companies.
Re:Malicious PyPI packages stealing credit cards and injecti... by tikkabhuna on Tuesday August 03, 2021 @09:54AM (#28049196)
We use an artifact server and our build servers are completely airgapped. We know exactly what dependencies are used across the organisation. We can take centralised action against malicious dependencies.

I wouldn't bother having one if you're small (<25) people. If you start having a centralised Infosec group, then it starts to become necessary.

Re:Malicious PyPI packages stealing credit cards and injecti... by njharman on Tuesday August 03, 2021 @10:40AM (#28049770)
Airgaped? Really? Everytime a build happens someone physically moves a Thu drive or some other media too from the build server?

Airgap means not networked, even internally. Not just "blocked" from internet.

Re:Malicious PyPI packages stealing credit cards and injecti... by tikkabhuna on Tuesday August 03, 2021 @11:59AM (#28050804)
Wikipedia[1] offers a slightly relaxed definition, although I agree, I (and my colleagues) abuse the term.

The artifact repository server connects to the internet via a proxy. Build servers have no access to the internet.

[1] https://en.wikipedia.org/wiki/Air_gap_(networking)

Re:Malicious PyPI packages stealing credit cards and injecti... by formerly_proven on Tuesday August 03, 2021 @11:11AM (#28050138)
Depending on what you're using for package management, an "artifact server" can be as simple as a directory in git or a dumb HTTP server. File names are non-colliding and you don't really need an audit log on that server, because all references are through lock-files in git with hashes (right? RIGHT?), so it basically doesn't matter what's on there.