Tell me you haven't written a large library for a wide array of users (many of which you wouldn't trust with a potato peeler) without telling me

Yes, it does matter even in functionally pure languages.

If the inner structure of your API's types are visible to callers, that's now part of your API, because callers can and will start to rely on that specific representation. Now you're constrained from updating your API, in order not to break code which depends on it.