News for hackers. Stuff that matters.
Re:Facebook hit git performance issue on large repository by groby_b on Friday February 03, 2012 @09:30PM (#3550022)
An inotify daemon could help, but it's not perfect: it needs a long time to warm up in the case of a reboot or crash

So does, presumably, the cache when you use lstat. (Let's scratch presumably. It does. Bonus points if you can't use Linux and use an OS that seems to chill its caches down as soon as possible. )

I hope I'm wrong, but the proper solution to this seems to be a custom file system - not only will it allow you to more easily obtain a "modified since" list of files, it also allows you to only get local files "on demand". (E.g. http://google-engtools.blogspot.com/2011/06/build-in-cloud-a...)

That still doesn't solve the data structure issues in git, but at least it takes some of the insane amount of I/O off the table.

I'm looking forward to see what you guys cook up :)

Re:Facebook hit git performance issue on large repository by caf on Saturday February 04, 2012 @12:08AM (#3550309)
You might be able to do the "custom file system" as a pass-through FUSE filesystem.
Re:Facebook hit git performance issue on large repository by patangay on Saturday February 04, 2012 @12:52AM (#3550404)
Yea, a FUSE filesystem is also being considered as one of the possible solutions.