Re:GitHub's new Git storage architecture
by jiggawatts on Saturday November 07, 2026 @03:43PM
(#50407866)
Am I the only one reading this with one raised eyebrow, wondering if a team of architecture astronauts have over-engineered this just a bit?
Pretty much everything in Git can be abstracted away as “immutable blob that can be conveniently content-indexed” and an “event log”. Both are embarrassingly parallelisable since repos are independent, and every blob is independent!
Caching is trivial over immutable blobs, and they would just need to partition the event log store (Kafka or whatever) so that each repo is maps to one partition.
Kafka can sink, what, a million events per partition per second?
Similarly, LSM log and KV storage like Microsoft FASTER would let them scale even that to about a hundred million events per sec, far beyond their target!
Pretty much everything in Git can be abstracted away as “immutable blob that can be conveniently content-indexed” and an “event log”. Both are embarrassingly parallelisable since repos are independent, and every blob is independent!
Caching is trivial over immutable blobs, and they would just need to partition the event log store (Kafka or whatever) so that each repo is maps to one partition.
Kafka can sink, what, a million events per partition per second?
Similarly, LSM log and KV storage like Microsoft FASTER would let them scale even that to about a hundred million events per sec, far beyond their target!
Am I missing something?