Scaling ChatGPT and scaling GitHub are very different problems.

For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.

GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.

> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.

Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.

And they have been working on this semi-shared state design for over 10 years now.

Every response from GitHub must be deterministic. OpenAI and Anthropic do not have this problem.

OpenAI, Anthropic, OpenRouter, Gemeni, etc also have another escape hatch: they can arbitrarily and potentially invisibly reduce your quality of service at their discretion. They can choose to route your request to an expensive model or a cheap one. By contrast, the best Github can do is slow your request down.

And they never even have to admit they do it.