Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have previously needed to externally fetch.

    bundle package

The problem then becomes the Gemfile and Gemfile.lock, which should really be in that submodule as well. You need to pass flags to bundler commands because it assumes the Gemfile is in the project root.

I think a full solution requires packaging, and using a modified buildpack that skips the bundle step.

Places the gem binaries in vendor/cache, as noted. SCM those.

"While installing gems, Bundler will check vendor/cache and then your system's gems. If a gem isn't cached or installed, Bundler will try to install it from the sources you have declared in your Gemfile."

http://gembundler.com/bundle_install.html

Heroku uses this tree lookup AFAIK.

UPDATE:

For others' edification, the default heroku ruby buildpack respects vendor/cache, but will purge it in the following scenarios:

  * if vendor/ruby_version exists
  * if vendor/heroku/buildpack_version exists, but vendor/heroku/ruby_version does not
  * if the bundler cache exists, but vendor/heroku/ruby_version file specifies a different version of ruby than the one actually being used.

To avoid packages sneakily trying to download their own dependencies from the internet we run pip install with a "--proxy http://localhost:9999 argument (where nothing is actually running on that port) so that we'll see an instant failure if something tries to pull a dependency over the network.

The non-existant proxy trick seems useful, I'll have to try that out.

Does anyone know why rubygems does not work this way? I had always just assumed it did (due to the historical intertwining of Ruby and Perl communities).

Personally I'm a big fan of the CPAN approach as it is fairly simply. Just mirror via FTP. It's a nobrainer to setup and run a mirror.

That said, CPAN's master (PAUSE.cpan.org) is a SPOF as well.

What I like is that not a single party is responsible for paying server bills + maintaining the platform. Ruby Central and the team of volunteers do a great job, but in the end, people only care when something breaks.

Instead every big company/university that profits from the Ruby ecosystem should imho run a public rubygems mirror as a contribution to the open source world. That's common practice for other projects, too. Think of all mirrors of the Linux distributions, kernel.org, cpan, python etc.

=> http://slideshare.net/rmoriz/rubygems-behind-the-gems

I also want to mention, that ftp.ruby-lang.org is a single homed box. There is no other official mirror of the MRI/C-Ruby source that can be used as failover or load balancer. This is bad, too.

Here's the docs that describe it:

"While installing gems, Bundler will check vendor/cache and then your system's gems."<and then tries to fetch remotely unless you pass --local>

http://gembundler.com/v1.2/bundle_install.html

(I miss the days from when github also hosted a gem repository…)

Solving the authenticity problem alone is probably not fun – tho obviously there is much to be learned from CPAN. Given recent problems there will probably be enough political will to make this happen in the future, though.

If you want to run your own PyPI internally, here's a very simple PyPI server (~150 lines of Python) that I wrote: https://github.com/steiza/simplepypi

What I've personally been looking for is an easy to setup caching proxy for PyPI. Something that is pip-compatible and serves files if it has them but will also fetch and then store packages if it doesn't. That way you could build up a collection of 3rd party packages over time, without having to explicitly manage it.

It probably wouldn't be hard to roll my own with a reverse proxy but it never gets moved to the front burner.

http://djangopypi2.readthedocs.org/en/latest/

For now though we'll probably just create a new git repo with a folder full of source distros (tarballs and zips), as mentioned above.

.NET developers, you can set up a similar cache for NuGet packages to avoid downtime (and reduce bandwidth usage): http://www.hanselman.com/blog/HowToAccessNuGetWhenNuGetorgIs...

  >  It's something that we in the Python community have already learned due to the historical unreliability of our equivalent package repo, PyPI.

Thanks for the perspective at any rate. Maybe too much coffee for me this morning? :)

rubygems.org is a central distribution platform trusted by tons and tons of projects. As such, that site is the one site you probably do not ever want compromized. Imagine the damage an attacker could deal by uploading backdoored versions of various popular gems.

I know - applying security patches is time-consuming and we are all afraid of breakage. But the moment rubygems.org stepped up to be a semi-official central distribution point for gems, I would have hoped they also took on the responsibility that goes along with that.

If this was some new unknown 0day exploit, I would be much more understanding, but this was known to exist, known to be dangerous, known to be exploited.

Gems, specifically should be signed. They are not, this type of exploit will continue to happen, hell, remember when github screwed up ssh keys? Who knows what's in the ecosystem.

TL;DR; Ruby's security ecosystem is butter.

Disclaimer: I love ruby and use it daily. It was two critical problems IMO: unsigned code and GIL. Yes, GIL. I'm looking at you ruby-core.

(This isn't a new technique -- for example, .deb packages distributed through APT are usually signed with gpg -- IIRC, this was a measure introduced years ago in response to a Debian mirror being compromised.)

Debian has (had?) a high barrier to entry to become a developer, and every developer signs their packages. The release binaries are arranged on a secured box and the release key itself is held by a limited set of people.

In short, the signatures work because of the human element and organizational structure of Debian.

Rubygems accepts submissions from the general public.

So, again, I don't see how it would have helped.

Talk about luck. :(

Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.

It's additional effort to deploy dangerous code.

Are the bigwigs going to authorize you time to look through all gems for potential backdoors or are they going to get i for free with their Heroku hosting?

Good to know someone is watching your back :)

I'm just thinking out loud.

I guess it would depend on the folks doing the investigation. If an exact timestamp could be determined for when things could have been compromised, you just roll back to a short while before that time.

I've been chastised before in rails irc for this. I strongly believe the source of all depndencies possible should be in your repository.

How far do you go? Do you include libxml for building nokogiri? Heck, do you include libc and gcc for building any gem with a C extension?

Coming from Java, Maven and something like Nexus Sonatype make it easy (for certain values of "easy") to run a proxy repository. The equivalent of all "gem install <some_gem>" goes through the proxy, which continues to serve gems even if the original source goes away.

I don't particularly like the inclusion of dependencies in a repository. Is this a custom version "some guy" long gone from the company created three years ago? Can I safely upgrade it to get security fix <X>? I suppose similar questions arise no matter the source...

This is reason why you cryptographically sign your gems before publishing them. I (unfortunately) had not known this was supported by RubyGems, but it is: http://docs.rubygems.org/read/chapter/21

But I'll bet very few gems are signed. Rails does not appear to be:

  $ gem install rails -P HighSecurity
  ERROR:  While executing gem ... (Gem::Exception)
    Unsigned gem

Today I use bundler but my solution back in the (java) day was to separate code releases from dependency releases (separate directories basically).

e.g.

    /path/2/project 
    /path/2/project-resources

I would not be surprised if we see even more of this as people feel out all of the other places that YAML is used as a user-facing data interchange format.

We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.

This type of proxy wouldn't help in this particular case but it would allow you to keep traffic to RubyGems.org down and also give you the ability to easily host private Gems.

You're probably aware, but it's possible to host private gems in a simple static webserver. We have our CI server copy gems over and run a script that calls "gem generate_index -d /path/to/gems".

Completely featureless, of course. Lacks the newer rubygems.org api (so bundler is stuck downloading the whole index), and obsolete versions will stick around without manual intervention (which, if you have too many, is a pathological case for bundler's resolver).

I generally try to follow the "vendor everything" philosophy: http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-a...

On a 27" monitor, it's almost like a child sized head right up in your face. I can only imagine it being even worse on bigger screens.

    document.getElementById('mugshot').remove()

What's more annoying: the odd (and wrong) belief that "An green apple" is grammatically correct. [1]

[0] http://english.stackexchange.com/questions/1016/do-you-use-a...

[1] http://english.stackexchange.com/questions/152/when-should-i...

[1] http://www.crowdhoster.com/

Does anybody have another suggestion for safely working around this issue? I don't have a clear sense for how long this will take to resolve and don't wish to slow down our release pace too much.

If rubygems.org is keeping fingerprints of each gem, then it still isn't sufficient, since those could have been compromised as well. If there's no other trustworthy source of fingerprints, then maybe we need to crowdsource it. Built a tool that will md5sum all the .gem files in your local cache directory, so that we can look for any files that were changed on rubygems.org

[1] http://docs.rubygems.org/read/chapter/21

When did the compromise happen? Was it compromised yesterday or only found out yesterday?

I have default gems installed on my system and haven't updated anything since the big Rails security issue that was reported a bit ago.

It'd be great to get some guidance on what to do.

I agree that the production/development split is not entirely clear without additional explanation. We've spent a lot of time thinking about how to communicate these things and have so far not come up with a way that we feel better describes the issues at hand.