The closest thing to a continuous-time RNN that exists are Neural Hawkes Processes [https://arxiv.org/abs/1612.09328]. They use a different model where observing the system necessarily changes its state, which is natural in some settings but doesn't fit in others. On the other hand, their model scales today and ours doesn't :)

For using a numerical solver e.g. Runge-Kutta you need nothing to master, statistics operations master.