That's a great question. I suppose they split it up into separate requests since downloading all of the dependencies at the same time might end up being slow and returning a lot of data, especially for something like Rails with a lot of dependencies. Or maybe separate requests was fast enough and a good trade off between speed/data and number of requests.

But bundler cannot determine which versions it needs until it has all dependencies, right? And in effect, all it does is recursively call for one "layer" of dependencies after the other, so it seems like the backend might just as well recursively query itself and return all dependencies at once.

as you have seen in the article, bundler needs dependencies of all versions as it might have to deal with older releases. Building the whole dependency graph for all versions of a gem with many dependencies in itself will a) take a very long time and b) probably be huge - likely much bigger than the whole gem index that was downloaded up until 1.1. The time required for parsing those MB after MB of data of which most will be thrown away will quickly get significant.

Another option would be to query for all dependencies of a specific version of a gem. That would reduce the amount of data, but it would still mean more work for the server and it would produce a lot more data to cache (much of which would never be queried more than once)

With the currently applied method, only the data that's really required must be queried and it can easily be cached on the server side too.

I suspect its a question of server resources. Dependency analysis is bundler's job and its better to let thousands of cpus run the algorithm. This leaves rubygems.org to simply service simple queries quickly without much load even though it means there will be many queries as part of each bundle.

We could try more than one layer. At the time we wrote it (last summer) the Bundler guys wanted to just start with one layer.