I'm thinking it might make sense to have a sort of hierarchy of models. The stupidest model predicts that everything will be stationary. The second model predicts that everything will travel in a straight line. The third model tries to predict a circular arc based on same fusion of path history and observed steering input. Fourth model uses a notion of action to predict whats going to happen, like "the car is changing lanes". Fifth model uses body language and common sense to predict intention "the pedestrian wants to cross".

Each model can potentially predict longer into the future but also has more complexity and things that can go wrong. So you keep track of how well each model is doing (on an object basis) and if one level is failing then you fall back on a stupider one. You might also want to increase caution if your models are not doing well (lower speed and increased safety distance).