Wide Neural Networks of Any Architecture Are Gaussian Processes

Re:Wide Neural Networks of Any Architecture Are Gaussian Pro... by credit_guy on Thursday November 28, 2019 @09:17AM (#21657499)

> Try throw me an architecture and watch if I can't throw you back a GP :)

On the pragmatic side, would that GP train faster than the NN? In my little experimentation with GPs, I found them awfully slow. However, maybe what I tried (it was black box for me) used some brute force approach, and there are other more fine-tuned algorithms. Since you are an expert in the area, what's your take?

[ Parent | on HN ]

Re:Wide Neural Networks of Any Architecture Are Gaussian Pro... by gyang on Thursday November 28, 2019 @06:39PM (#21660888)

I think in general, "training" a GP, i.e. doing GP inference (or kernel regression) is not done for speed reasons, but rather because they are sample efficient. More concretely, the practical folk wisdom regarding GPs is that when there are not much data, then GP inference with a well-chosen kernel can give you much more bang for the buck than a neural network. However, when there are a lot of data (especially in the perceptual domains like vision and language), neural networks typically train faster and generalize better.

I wouldn't say I'm an expert at using GPs, so actual GP practitioners feel free to correct me if I'm wrong :)

[ Parent | on HN ]