I don’t think image+text pair models (E.G. CLIP) would be very useful for AV tasks. They’re not very good at fine-grained classification or counting the number of instances in a scene. Not even getting into latency or model size.

8+ responding distance dots that are much closer than they should be are already a STOP, APPLY BREAKS situation where not much ML is even needed. That's the beauty of LIDAR. It will save lives unlike Tesla's driving into obvious poles/truck booms mistaken for a statue of liberty in Nebraska.

To what degree can current ML models and techniques for self-driving understand what's happening based on data captured at multiple time points, maintaining object permanence?

For example, if a a car is stopped at a red light and it sees a child across the street running and the child runs behind a truck it will take her 5 seconds to get past at her last observed rate of speed, and the light turns green at a moment when the car would hit the child as she emerges from behind the truck (given the preceding) as it drives by the truck, do any ML systems have the ability to predict this?

People would likely be cautious if they saw the same scene, and would approach slowly. I don't think that current ML systems are able to do that, but would like to hear some informed opinions.

So is this a self-fulfilling prophecy? Some guy decides a trillion dollar company should discard an entire dataset for no reason, and as a result a decade late the company can finally actually do the job, thus proving the guy ?right?

At least with radar, it was causing the car to make more mistakes given it was reacting to either.

How much of that camera data is labeled for distance measurement use cases?

Because cars (generally) move, probably most of it. You can compare successive frames and make predictions.

I think Tesla actually used radar data to provide ground truth for this. So they don't even do that to generate ground truth. What happens if things have the same relative speed when doing the labeling? Or low light conditions? How do you account for per part variation in lens and sensor designs and how it messes with your predictions?

Am I missing something? What specifically is a paradigm shift in the last three months?

Elsewhere in this subthread was mentions CLIP[0]

About 2/3rds down the page features 'Object tracking' and 'Robotic control'

[0] https://blog.roboflow.com/openai-clip/

This isn't something from the past three months...