Is there any point where you would want to pass a pointer but NOT because you want mutable state, simply because it's more efficient or something?

Well, pointers have fixed size, random structures may be arbitrarily big and involve a crapload of copying.