Re:Incident Report: Inadvertent Private Repository Disclosure
by foota on Friday October 28, 2016 @03:34PM
(#12819843)
I'm curious, how did they calculate entropy? My first thought was to do something with Huffman encoding.
Re:Incident Report: Inadvertent Private Repository Disclosure
by jasonmoo on Friday October 28, 2016 @03:48PM
(#12819975)
I wrote the script in question and actually used a simple shannon entropy value. (http://codereview.stackexchange.com/questions/868/calculatin...). It worked well enough help rule out several problem spaces.
Re:Incident Report: Inadvertent Private Repository Disclosure
by pavel_lishin on Friday October 28, 2016 @04:23PM
(#12820269)
Would you mind posting the script? I'd love to run it against our codebase and see what it comes up with.
It might be a fun thing to open source as part of a "I've inherited a project, what now?" toolkit that helps you decide what to fix.
Re:Incident Report: Inadvertent Private Repository Disclosure
by jasonmoo on Friday October 28, 2016 @04:43PM
(#12820488)
Sure. It's a simple tool but the concept could be augmented toward something like the scenario you described.
https://gist.github.com/jasonmoo/06691c8fea09b62aa35235fc93e...
Re:Incident Report: Inadvertent Private Repository Disclosure
by ubernostrum on Friday October 28, 2016 @07:32PM
(#12821664)
IIRC Instagram released a plugin for Bandit (the OpenStack static analyzer for Python) that does this.
Re:Incident Report: Inadvertent Private Repository Disclosure
by dorianm on Friday October 28, 2016 @08:01PM
(#12821791)
I thought the idea was interesting so here is a little PoC in Ruby:
require 'facets'
lines = Dir['**/*.rb', '**/*.py', '**/*.cpp'].map { |f| File.read(f).lines.map(&:chars) }.inject(&:+)
puts lines.sort_by(&:entropy).map(&:join).last(10).reverse
Using: http://www.rubydoc.info/github/rubyworks/facets/Array%3Aentr...
Re:Incident Report: Inadvertent Private Repository Disclosure
by gcr on Friday October 28, 2016 @08:28PM
(#12821899)
Is it easy to modify this script to run over all lines that have ever existed in the repository history?
For example, could you pipe the output of `git log -p --all` through this and filter out all the commit hashes somehow?
Re:Incident Report: Inadvertent Private Repository Disclosure
by dorianm on Friday October 28, 2016 @09:21PM
(#12822098)
Yup, just use:
lines = `git log -p --all`.lines.map(&:chars)
So I found that `git grep /.+/ $(git rev-list --all)` is a better way to get the content of all the files: https://gist.github.com/Dorian/e1514535c3c5036cf327ce61eb34a...But actually an hex number regexp might me far more accurate than the entropy (e.g.: secrets are often long hex numbers).
I tried it and it yields interesting results: https://gist.github.com/28110f0b8105db11e8973d1d0be85259
Re:Incident Report: Inadvertent Private Repository Disclosure
by tw90210 on Friday October 28, 2016 @07:47PM
(#12821725)
I got the other end of that email today, saying that an account in my organization had inadvertently downloaded private repos from another customer when fetching from one of our own. Fortunately for GitHub/that user, it was almost certainly our automated provisioning system so we never had any idea and whatever it was never made it anywhere interesting.
The email was kind of funny though, part of it was effectively "if you have this data pretty please delete it without looking at it". I'm sure that's the best they can do, but it still made me chuckle.
Re:Incident Report: Inadvertent Private Repository Disclosure
by tonyarkles on Friday October 28, 2016 @08:46PM
(#12821976)
Of the very small number of repos affected, there are now two of us reporting that it affected us :). And I had the same approximate response: well, I don't keep secrets in the repository, so not that big of deal. I'd rather the source not get shared with the world, but shit happens and they owned up to it right away. If that source were valuable enough, I'd be hosting it on-prem with encrypted off-site backups.
One of our engineers came up with a useful script to grab all unique lines from the history of the repository and sort them according to entropy. This helps to lift any access keys or passwords which may have been committed at any point to the top.
I think this is a great example to illustrate the tough edges of security to less experienced engineers. Github will most likely never let something like this happen to you, but on the off-chance that they do it's great to be prepared. Additionally, the response from Github was very well received. No excuses, just a thorough explanation of what happened.
I also can't help but mention that we're hiring, if you'd like to work at an organization that values security and data privacy very highly. :) usebutton.com/join-us