News for hackers. Stuff that matters.
Re:Exploding Git Repositories by warent on Thursday October 12, 2017 @01:04PM (#15459450)
Odd. It's surprising to me that this example runs out of memory. What would be a possible solution?

Admittedly I don't know that much about the inner-workings of git, but off the top of my head, perhaps something with traversing the tree depth-first and releasing resources as you hit the bottom?

Re:Exploding Git Repositories by ericfrederich on Thursday October 12, 2017 @04:39PM (#15461129)
You need a problem to have a solution to it. What do you consider to be the problem here?

This is essentially something that can be expressed in relatively few bytes that expands to something much larger.

Imagine I had a compressed file format for blank files "0x00" the whole way. It is implemented by writing in ascii the size of the uncompressed file.

So the contents of a file called terrabyte.blank is just ascii "1000000000000" ... or the contents of a file called petabyte.blank is "10000000000000"

I cannot decompress these files... what is the solution?

Re:Exploding Git Repositories by geezerjay on Friday October 13, 2017 @03:46AM (#15463872)
>You need a problem to have a solution to it. What do you consider to be the problem here? > >This is essentially something that can be expressed in relatively few bytes that expands to something much larger.

That seems to be the problem. I mean, if an object expands to something much larger to the point that it crashes services just by the sheer volume of the resources it takes... That is pretty much the definition of an attack vector of a denial-of-service attack.

Re:Exploding Git Repositories by TeMPOraL on Friday October 13, 2017 @09:33AM (#15465441)
There is a problem here, but it's not with data. It's with the service.

Being able to express trees efficiently in a data format is an useful feature, but it requires the code processing it not to be lazy and assume people will never create pathological tree structures.

Re:Exploding Git Repositories by warent on Thursday October 12, 2017 @05:16PM (#15461395)
I'm not following; why can't you decompress it? Of course you cant decompress it into memory, but if it's trying to that then there's a problem in the code (problem identified).

Naive solution, just write to the end of the file and make sure you have enough disk. More sophisticated solution, shard the file across multiple disks.

Re:Exploding Git Repositories by Piskvorrr on Saturday October 14, 2017 @09:53AM (#15472169)
That's not a solution, that's sweeping the problem under the rug: "just have the OS provide storage, therefore it's not my problem any more, solved. (Never mind that with a few more layers, the tree would decompress into a structure larger than all the storage ever available to mankind)"
Re:Exploding Git Repositories by peff on Thursday October 12, 2017 @02:05PM (#15459888)
Git assumes it can keep a small struct in memory for each file in the repository (not the file contents, but a fixed per-file size). This repository just has a very large number of files.
Re:Exploding Git Repositories by glandium on Friday October 13, 2017 @02:47AM (#15463685)
Large as in 10 billions. Even if git only needed 1 byte in memory per file, it would need 10GB.
Re:Exploding Git Repositories by koenigdavidmj on Thursday October 12, 2017 @07:24PM (#15462172)
One option is to modify each of the utilities so that it doesn't have a full representation of the whole tree in memory. I doubt this is feasible in all cases, though for something like 'git status' it should be doable.

If the tree object format was required to store its own path, then you wouldn't be able to repeat the tree a bunch of times. The in-memory representation would be the same size, but you would now need that same number of objects in the repository. No more exponential fanout.

But that would kind of defeat the purpose of Git for real use cases (renaming a directory shouldn't make the size of your repo blow up).

Re:Exploding Git Repositories by TeMPOraL on Friday October 13, 2017 @09:28AM (#15465404)
Have git (the client) monitor its own memory usage and abort if it gets above a set limit (say, default, 1GB), with a message that tells you how to change or disable the limit.