Wow this is awesome! Thanks a ton for taking the time to think about the project and post this! Yeah I think the way to go is:

1. Improve the model input by stacking frames 2. Then try model distillation to a smaller model