Hi Radford, I'm very happy that this paper got your attention, but I'm regretful that I did not represent your research accurately!

Indeed, in your thesis, section 2.3 is on "Priors for networks with more than one hidden layer." I will fix this in the next version of the paper, and as a prerequisite, I'd like to make sure I understand your contributions correctly.

Is the following summary accurate?

In section 2.3, you explored the GP limit for more than 1 hidden layer numerically, as well as some thoughts on the decay behavior of the GP kernel associated to an MLP with step function nonlinearity. You also considered mixing Gaussian and non-Gaussian priors in different hidden layers and finally mused about the infinite depth limit of the infinitely-wide MLP.

However, I could not find a rigorous treatment of the multi-layer GP limit (in the vein of Lee et al. and Matthews et al. (2019)). Does it exist elsewhere in the paper?

Looking now at my thesis, I agree that I don't explicitly argue theoretically for why multilayer networks will (under suitable conditions) converge to Gaussian processes. However, it follows (at some level of rigour) pretty directly from the fact (which I do note) that if a single-hidden layer has multiple outputs, the functions computed by these outputs will be independent (in the prior) as the number of hidden units goes to infinity. So if you add another hidden layer, the functions computed by the units in this layer will be independent (they're like outputs of the previous layer), and the argument for why the outputs from this layer form a GP goes through as before. I'm not sure why I didn't explicitly note this. It's implicitly assumed in my discussion of how the covariance function for networks with step function hidden units changes as you add more layers.

Right, so your argument would work if you allow the layer widths to tend to infinity sequentially (so this corresponds to finite networks where each previous layer is much bigger than the next layer). This is the argument presented by Lee et al. But note it's nontrivial to argue that this limit holds when the widths of all layers tend to infinity at the same time (arguably the more natural limit), which is one of the main contributions of Matthews et al. In my paper here, I also consider this limit where the widths tend to infinity at the same time.

In any case, I'll update the paper to reflect our discussion here. Thanks, Radford!