We looked into Git Ketch https://www.infoq.com/news/2016/02/google-kick-starts-git-ke... for a distributed system based on hooks. But this requires a sync per repository (very slow with 100k+ repos per server) and it is no longer actively being worked on by Google.
There are plenty of off-the-shelf consensus algorithms/services you could use in the git-hooks that don't have the git-ketch limitation. Zookeeper, etcd, consule, redis, etc. It really just matters your criteria and requirements, they all of trade-offs but I'm sure one fits.
To be frank I think your time is better spent looking into why one of these consensus packages isn't the right call for your needs. Since they could easily plug into the git-hooks & git-shell commands, it could avoid a massive investment of time from your team and you.
A little while ago I spent a night and I was able to get a fully distributed git http server up and running. I used HBase and was able to get it most things working. That allows the only contention to be around refs. Everything else is well distributed and spread across as many machines as you need. Caching is handled on the key value store side, and the http server side. I didn't get to git gc work though.
This is solved without modifying the git client using a distributed/shared lock wrapping the git-shell & git-hooks (pre-receive) . If you were to modify the git server behavior to say use epaxos, one could provide even better latency guaranties https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf .
I'm not sure about the GC operation though, did you use the DfsGarbageCollector that is part of Jgit? I thought that was provided as part of the DFS interface.
http://download.eclipse.org/jgit/docs/jgit-2.0.0.20120613090...
The way GitLab is organized (pre-Gitaly) makes it very hard to do things like that. Roughly speaking, when handling a push, it is a coordinated dance between GitLab components to make sure it all goes in, but none of these components has full control over the push from start to finish. One of the reasons we are creating Gitaly is to create a 'place' (namely Gitaly) where we _can_ exercise such control.
The other big reason for building it is not having to use Git repos on NFS.
(Gitaly maintainer)
My query being you can effectively use the native pre-receive hooks & wrap the git shell to allow consensus without needing to replace git with an RPC system like Gitaly.