Neural Ordinary Differential Equations

Re:Neural Ordinary Differential Equations by duvenaud on Friday December 14, 2018 @12:49PM (#18682790)

The comparison we make is to residual networks, which I think is valid. First, we do parameterize a theta that changes with time, using a hypernet. But this is equivalent to the way sampo wrote the equations above - you can just feed time as another input to the dynamics network to get dynamics that change with time.

Second, I agree that general feedforward nets allow dimension changes, but resnets don't. This model is a drop-in replacement for resnets, but not for any feedforward net. If we gave the wrong impression somewhere, please let us know.

We didn't make the analogy with RNNs, because I don't think it fits - standard input-output RNNs have to take in part of the data with every time step, while here the data only appears at the input (depth 0) layer of the network.

[ Parent | on HN ]

Re:Neural Ordinary Differential Equations by soVeryTired on Friday December 14, 2018 @07:45PM (#18686268)

You're absolutely right - sorry, somehow I managed to miss the explicit time parameter in your equation two, and didn't read carefully enough to see that you were restricting the discussion to resnets and normalising flows.

You might be able to make a better connection to RNNs by having the input data as a 'forcing' function in your ODE. But you probably need some regularity conditions on the input data to make sure the result is nicely behaved.

[ Parent | on HN ]

Re:Neural Ordinary Differential Equations by im3w1l on Saturday December 15, 2018 @01:48AM (#18687398)

> standard input-output RNNs have to take in part of the data with every time step

Well they can but I don't see they why they have to. And couldn't your network also take input and give output for all times?

> First, we do parameterize a theta that changes with time, using a hypernet.

Ah I see. Did you end up using it in the final model? I don't see that in the mnist example, but I could be missing it as I only skimmed the code.

[ Parent | on HN ]

Re:Neural Ordinary Differential Equations by joe_the_user on Friday December 14, 2018 @01:08PM (#18682984)

Hasn't there been similar work to this in the past?

I don't see a "related work" section in your paper.

[ Parent | on HN ]

Re:Neural Ordinary Differential Equations by duvenaud on Friday December 14, 2018 @06:56PM (#18686007)

Section 7 is titled "Related Work".

[ Parent | on HN ]

Re:Neural Ordinary Differential Equations by joe_the_user on Friday December 14, 2018 @07:55PM (#18686302)

Thanks,

Apologies for not looking harder

[ Parent | on HN ]