News for hackers. Stuff that matters.
Re:Rubygems.org compromised by qrush on Wednesday January 30, 2013 @11:16AM (#5140109)
Hi everyone -

A full post-mortem is coming, but here's what happened from my perspective. I got mentioned on twitter about this thread while I was on the bus. I asked @evanphx to put the site into maintenance mode immediately.

Once I got on wifi, I downloaded the "exploit" gems and found that they used the Psych hack to post config/initializers/* and config/*.yml to Pastie. Luckily, none of our API keys were actually in those files - our config is a bit different since you can still run RubyGems.org on Heroku, which needs to put secret keys in the ENV.

I deleted the exploit gems permanently along with the throwaway account that posted them...that's why this post's URL is a 404.

I've reset our S3 keys just to be sure and any other keys we had on our production site. From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure. Once we get a real fix for this issue out, pushing gems will be enabled again.

Just a general PSA -

Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.

Re:Rubygems.org compromised by tptacek on Wednesday January 30, 2013 @12:12PM (#5140470)
If you want security notices to go to the Rubygems team, you have to set up a security page that tells people how to do that. Like everyone else, I appreciate your volunteer work, but no amount of goodwill creates the ability for people to read your mind.

Please post a security page. It literally doesn't need anything more than an email address and a PGP public key.

By the way: if Rubygems needs security help, it looks like there's quite a number of people who are willing to pitch in. When I published a FreeBSD crt0 bug back in 1996, I was given commit privs. I thought that was a pretty effective way to co-opt adversarial researchers.

Re:Rubygems.org compromised by qrush on Wednesday January 30, 2013 @12:14PM (#5140480)
Great suggestion, I'll add a security page today.
Re:Rubygems.org compromised by tptacek on Wednesday January 30, 2013 @12:29PM (#5140580)
Great!

I know dealing with this stuff is no fun. Try to keep in mind, though, that people like Ben Murphy and Hal (Postmodern) are on your side. Again, maybe consider formalizing that relationship a little!

Re:Rubygems.org compromised by jasonkester on Wednesday January 30, 2013 @12:32PM (#5140606)
What would the PGP key be for?
Re:Rubygems.org compromised by tptacek on Wednesday January 30, 2013 @12:34PM (#5140614)
The Rubygems team (or even just the person who writes the security page) can just generate a new one, for "[email protected]" or whatever, and it can be shared by the team.

The purpose of the key is to allow people to report security vulnerabilities without worrying that by doing so they're giving ammunition to people snooping emails.

Re:Rubygems.org compromised by willyt on Wednesday January 30, 2013 @02:19PM (#5141191)
Maybe I dont understand PGP and this is a stupid question, but if the site is compromised, would it not be possible for them to just put a different key up? One which they knew the private key for? How would you know the public key on the compromised site has not been changed unless you could compare it to the PGP key from before the site was compromised?
Re:Rubygems.org compromised by rmoriz on Wednesday January 30, 2013 @03:12PM (#5141490)
Re:Rubygems.org compromised by tptacek on Wednesday January 30, 2013 @02:38PM (#5141284)
Publishing the key is what allows you to detect malicious changes to the key.
Re:Rubygems.org compromised by JoachimSchipper on Wednesday January 30, 2013 @02:32PM (#5141257)
You can't, but having a security.html page gives attackers the possibility to contact you privately and securely; this reduces the chance that they'll decide to "contact" you by completely pwning your public site.
Re:Rubygems.org compromised by arjie on Wednesday January 30, 2013 @02:46PM (#5141344)
What if the key is endorsed by the individual people on the team? Their personal keys, which they keep, must sign the security team key. A replacement cannot have this endorsement.
Re:Rubygems.org compromised by jaytaylor on Wednesday January 30, 2013 @02:40PM (#5141299)
@JoachimSchipper do sec people inherently know to try hitting /security.html by convention, or should there be links to it from the main site?
Re:Rubygems.org compromised by tptacek on Wednesday January 30, 2013 @02:48PM (#5141362)
They know to look for a link to the security page.

Github's is in their site footer.

37signals' is in their site footer.

Twitter's is linked off the sidebar in their "About" page.

Google's and Facebook's are the top search result for their site and "vulnerability" "security".

These are all fine options.

Re:Rubygems.org compromised by jaytaylor on Wednesday January 30, 2013 @04:39PM (#5141916)
Thank you!
Re:Rubygems.org compromised by kgo on Wednesday January 30, 2013 @12:35PM (#5140619)
You don't want to send an actual exploit in clear text to a possibly compromised email account. PGP encrypts, and provides some level of validation that the message can only be read by someone on the security team.
Re:Rubygems.org compromised by jazzyb on Wednesday January 30, 2013 @12:36PM (#5140626)
For encrypting the email with the exploit details. This is quite a common practice in the security industry.
Re:Rubygems.org compromised by zobzu on Wednesday January 30, 2013 @02:16PM (#5141161)
Like signing gems? cough
Re:Rubygems.org compromised by i386 on Wednesday January 30, 2013 @03:44PM (#5141657)
Signing gems should be mandatory
Re:Rubygems.org compromised by zobzu on Wednesday January 30, 2013 @09:38PM (#5143090)
Don't get me started on python eggs :)
Re:Rubygems.org compromised by homakov on Wednesday January 30, 2013 @04:58PM (#5142028)
i hope there is security@rubygems
Re:Rubygems.org compromised by ig1 on Wednesday January 30, 2013 @11:24AM (#5140165)
Respectfully, RubyGems has already been compromised. Not telling anyone has zero advantages to RubyGem users.

Telling people to avoid using RubyGems and to audit if they've used it recently helps protect users.

Responsible disclosure is about avoiding giving bad guys new attack vectors and ensuring that the maintainer has the best chance of fixing the issue before a blackhat discovers and exploits it. That doesn't apply in this case.

Re:Rubygems.org compromised by benatkin on Wednesday January 30, 2013 @12:13PM (#5140476)
I think you meant to say "respectfully".

I agree, but how do people audit? I think it's time to start putting application dependencies, like rubygems and npm modules, under version control, not in the same repo, but in a different repo.

Mikeal Rogers has some good reasons why, but there gets to be too much commit noise if you have big dependencies, and put them in the same repo. A separate repo solves this.

http://www.futurealoof.com/posts/nodemodules-in-git.html

Re:Rubygems.org compromised by ig1 on Wednesday January 30, 2013 @12:55PM (#5140727)
Yes I did, fixed.
Re:Rubygems.org compromised by tangue on Wednesday January 30, 2013 @11:54AM (#5140352)
Totally agree. I know it's a pain for the admins, but you can't say something like "security is omakase."
Re:Rubygems.org compromised by blambeau on Wednesday January 30, 2013 @11:24AM (#5140157)
> Please, if you find an issue like this, be nice. Tell the maintainers privately.

Nick. I dont support today's PoC, I don't, really. But I told you. Twice in a week. I feel deeply sad now.

Re:Rubygems.org compromised by qrush on Wednesday January 30, 2013 @12:52PM (#5140706)
Yes, and thanks for telling us. Really, we should have disabled gem pushes immediately. Hindsight is 20/20, and I'm not sure why I didn't think of doing so earlier. The pain of not being able to push gems would have forced us to fix it.

I'm sorry about this. I don't know what else to say. I wish we didn't have to deal with this kind of problem in the Ruby community.

Re:Rubygems.org compromised by blambeau on Wednesday January 30, 2013 @11:27AM (#5140178)
But thanks for your work anyway.
Re:Rubygems.org compromised by username3 on Wednesday January 30, 2013 @11:27AM (#5140182)
Proof of Concept?
Re:Rubygems.org compromised by ShawnBird on Wednesday January 30, 2013 @11:49AM (#5140325)
Point of Contact
Re:Rubygems.org compromised by bradly on Wednesday January 30, 2013 @11:25AM (#5140167)
> From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure.

Do you think it makes sense to disable pulling down gems until you know for sure nothing in S3 is compromised?

Re:Rubygems.org compromised by tjic on Wednesday January 30, 2013 @11:36AM (#5140241)
"THIS!", as the kids say.
Re:Rubygems.org compromised by akl on Wednesday January 30, 2013 @12:04PM (#5140430)
> Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.

It'd probably go a long way if you guys considered making an easily-discoverable security page with information on who to contact, how you prefer that contact to take place and how long they should expect to have to wait to hear back from you.

Re:Rubygems.org compromised by purephase on Wednesday January 30, 2013 @11:26AM (#5140173)
Is this maybe an opportunity to re-evaluate the volunteer only aspect of RubyGems? I love the service (as I'm sure thousands of others do) and I'd happily pay for a subscription model that would fund a full-time developer/maintainer for the service. Based on how critical it is to the Ruby community I can't imagine that I'm alone in thinking this.

Just a suggestion!

Re:Rubygems.org compromised by qrush on Wednesday January 30, 2013 @11:30AM (#5140194)
I am not personally interested in doing this. Nothing is stopping others, though: http://www.gemfury.com/
Re:Rubygems.org compromised by Xylakant on Wednesday January 30, 2013 @11:59AM (#5140391)
Hmm. That's for private gems. Interesting, but a different use-case. I'd chip in to pay for someone to maintain "rubygems.org" on a professional basis. It's awesome that some people take their private time to support such an essential service, but I'm with the grandparent that it might be good if it were run on a paid-for basis. Maybe through donations or similar, so that access is still free.
Re:Rubygems.org compromised by petercooper on Wednesday January 30, 2013 @12:16PM (#5140496)
Anyone know how CPAN works in this regard?
Re:Rubygems.org compromised by perigrin on Wednesday January 30, 2013 @01:19PM (#5140879)
CPAN is all volunteer. We happen to have a few really good SAs volunteer to keep core infrastructure up (which PAUSE, our upload server, just moved onto) ... but it's still volunteer.
Re:Rubygems.org compromised by Xylakant on Wednesday January 30, 2013 @01:28PM (#5140919)
How do you manage critical stuff like security problems such as this one? Do you have a process for that or is it best effort?
Re:Rubygems.org compromised by amy_seqmedia on Wednesday January 30, 2013 @11:24PM (#5143429)
Maybe funds support the RubyGems maintainers and the gem authors?
Re:Rubygems.org compromised by asdf333 on Wednesday January 30, 2013 @08:20PM (#5142806)
Thank you to you and all the volunteers who maintain Rubygems. Its an amazing repository that has helped all ruby developers.

Its amazing to see that one can just type 'gem blah' and magically things get installed.

Re:Rubygems.org compromised by neya on Wednesday January 30, 2013 @11:54AM (#5140351)
Hi Nick, instead of being a dick, I want to first of all thank you for understanding the urgency (despite being on a public bus, you took the pains to get it temporarily fixed somehow). Like others, I do agree that posting on HN does more good than bad, but at the same time, I do understand that it is also polite to inform the maintainers too (which I think in your case they didn't.). Anyway, thank you for being a responsible maintainer!

Once again, Thank you for maintaining rubygems.org! You're doing a great job :)

Cheers,

Neya

Re:Rubygems.org compromised by the-come-ons on Wednesday January 30, 2013 @11:57AM (#5140377)
Who is Amanda? I think you mean Nick...
Re:Rubygems.org compromised by neya on Wednesday January 30, 2013 @12:07PM (#5140443)
Yep, sorry :D changed it matey! Thx!!
Re:Rubygems.org compromised by wyuenho on Wednesday January 30, 2013 @12:15PM (#5140484)
Maybe it's time to heckle some sponsors (cough heroku cough) to pay for 2 full time admins for this critical piece of infrastructure.
Re:Rubygems.org compromised by marcgg on Wednesday January 30, 2013 @12:24PM (#5140553)
Thanks for handling this issue & your work on the site. I look forward to read the post mortem.

Oh, and +1 for the security page

Re:Rubygems.org compromised by arunagarwal on Wednesday January 30, 2013 @11:24AM (#5140163)
Thanks for all your work Nick!