Well, if there's a 1-1 map between GP and NNs, shouldn't we be able to determine the effect of a gradient descent step on a GP by combining the two maps?

I have only barely glanced at the paper mind-you, so I couldn't say the details but still.