Change up the model. That allows it to see previous frames and 1-2 future frames.

Then train the model on movies that are color and then turn them black and white.

That way you can train temporal coherence.