That's absolutely right, but it's not just large objects. Any situation where you follow links rarely compared to your other memory accesses then the cache performance of links becomes negligible and the benefit of simplicity and O(1) operations can shine. Vectors for scrubbing through things without changing their structure, lists and trees the opposite.

Scrubbing through things quickly is empirically more common, hence the standard advice to use vectors by default.