Sure it is. Whether it's an adequate model for some particular purpose is another question. But no model includes everything.
Objects and velocities would be a good start. I don't know enough about the present state of the technology to know whether we even have that. But from some of these accidents we're seeing, it seems like we don't.
I would agree that a better model would attempt to infer intentions: what is that driver/cyclist/pedestrian likely to do next, based on what they're doing now?
Current tech builds a (statistical) model of some aspects of the world that an agent must navigate- aspects that pertain to entity types, trajectories, velocities and so on.
But other bits are missing and they are important ones- for example, what is a human doing on top of a bicycle moving at the same speed as him or her? What is the relation between a driver and a car? And so on. We can still not represent stuff like that with any accuracy to speak of.
In short, what's missing for the most part is a representation of the relations between entities in the world- even where we can accurately represent the entities themselves, or their characteristics.
I think we humans are so good at just passively keeping track of everything in our environment we forget how difficult it is "keep a world model" from what one sees.