Do you want to return a copy/clone of the inner object or just let people mutate it, destroying all of your data structure's invariants? Yes, if you change the implementation of the structure, user code will also break and it is certainly easier to do that behind a method, this is indeed rare in practice but the nuisance of migrating direct access to indirect access is bigger than the nuisance of (today's) unnecessary indirection.
Since in some cases safety around invariants and future proofing will require the level of indirection, it is easier to just expect the convention. Moreover, codegen can just produce them for you, they can be excluded from test coverage. Then there are languages like python that allow you in the future to pretend that indirect access is direct access and cannot protect invariants in any case, so just go direct from day one.
All the hoops for this... Codegen, excluding from coverage, Lombok, and all that magic to hide you from a simple obj.field that is all you're doing in 99.9% of cases. It's just so rare in practice. I think the language itself should allow for readonly fields and property setters for the extremely rare occasions where you need this.