Transformers don't feel differentiable (because of the attention mechanism), but they actually are (as being back-propagation based forces it to be).

The attention mechanism is not a stretching of the manifold, but is trained to be able to measure distances in the manifold surface, which is stretched and deformed (or transformed?) in the feed-forward layers.