Yes we care, they are called references.

Use too much of them without thinking and performance goes down the drain thanks to cache misses.

This is why managed languages that care about performance have value types and padding support. Even Java is getting them.