Very similar experience here. Also just recently I really _had_ to use and extend the "internal" part of a legacy library. So potentially days or more than a week of work turned into a couple of hours.
Take something as simple as a vector (eg. std::vector in C++). If a user directly sets the size or capacity, the calls to methods like push_back() will behave incorrectly, or may even crash.
Opaque pointers are one way of hiding representation, but they also eliminate the possibility of inlining, unless LTO is in use. If you have members that need to be accessible in inline functions, it's impossible to use opaque pointers.
There is certainly a risk of "implicit interfaces" (Hyrum's Law), where users break even when you're changing the internals, but we can lessen the risk by encapsulating data structures as much as possible. There are other strategies for lessening this risk, like randomizing unspecified behaviors, so that people cannot take dependencies on behaviors that are not guaranteed.
You can, just not in the "strictly technical" sense. You add a "warranty void if these fields are touched" documentation string.
I write a lot of rust. By default, fields are private. It's rare to see a field in my code that omits the `pub` prefix. I sometimes start with private because I forget `pub`, but inevitably I need to make it public!
I like in principle they're there, but in practice, `pub` feels like syntactic clutter, because it's on all my fields! I think this is because I use structs as abstract bags of data, vice patterns with getters/setters.
When using libraries that rely on private fields, I sometimes have to fork them so I can get at the data. If they do provide a way, it makes the (auto-generated) docs less usable than if the fields were public.
I suspect this might come down to the perspective of application/firmware development vice lib development. The few times I do use private fields have been in libs. E.g. if you have matrix you generate from pub fields and similar.
> You cannot reasonably form API contracts (which are the foundation of software modularity) unless you can hide the internal representation. You need to be able to change the internal representation without breaking users.
You never need to hide internal representations to form an "API contract". That doesn't even make sense. If you need to be able to change the internal representation without breaking user code, you're looking for opaque pointers, which have been the solution to this problem since at least C89, I assume earlier.
If you change your data structures or the procedures that operate on them, you're almost certain to break someone's code somewhere, regardless of whether or not you hide the implementation.