They do. Karpathy spoke about it in on Tesla AI day. They use it for transforming image space to a vector space.

See: https://youtu.be/j0z4FweCy4M (timestamp 54.40 onwards)

Can you provide a more specific timestamp? 54.40 doesn't seem to mention anything about transformers, and "onwards" is two hours.

I'd be really surprised if they use transformers due to how computationally expensive they are for anything involving vision.

EDIT: Found it. 1h: https://www.youtube.com/watch?v=j0z4FweCy4M?t=1h

Fascinating. I guess transformers are efficient.

I gave the timestamp where they start talking about the problem they are trying to solve using transformers. As you said it is around 1hr mark

Doubtful. The biggest downside of transformers for vision is how ungodly long they take to produce results. Tesla has to operate in realtime.

In Karpathy’s recent AI day presentation he specifically stated they use transformers.

But not on the raw camera input — they use regnets for that. The transformers come higher up the stack:

https://youtu.be/j0z4FweCy4M

Transformers mentioned on the slide at timestamp 1:00:18.

They use the key-value lookup/routing mechanism from Transformers to predict pixel-wise labels in bird view (lane, car, obstacle, intersection etc.). The motivation here is that some of the predictions may temporarily be occluded, so for predicting these occluded areas it may be particularly helpful to attend to remote regions in the input images which requires long-range dependencies that highly depend on the input itself (e.g. on whether there is an occlusion), which is exactly where the key-value mechanism excels. Not sure they even process past camera frames at this point. They only mention that later in the pipline they have an LSTM-like NN incorporating past camera frames (Schmidhuber will be proud!!).

Edit: A random observation which just occurred to me is that their predictions seem surprisingly temporally unstable. Observe, for example, the lane layout wildly changing while it drives makes a left-turn at the intersection (https://youtu.be/j0z4FweCy4M?t=2608). You can use the comma and period keys to step through the video frame-by-frame.

Thank you!