I assume Insta invests in engineering for whatever cache expiration they do. And if there's 500MB cache, probably it's a conscious engineering decision.
But maybe the majority app developers, if they could fill up cache, probably aren't thinking that far ahead -- not thinking about that non-visual, non-customer-story aspects. So most wouldn't have occasion to learn how to do cache management sufficiently well that they don't cause stability or performance problems.
But hey, if the big names don't want to deal with it, more room for smaller teams.
I think there could be a real watershed moment in a few years as companies realize that rising to the technological challenge isn't actually that expensive or even difficult if you hire people with the right combination of arrogance and brains. Rage against the latrine.
Kind of a bummer I don't know of any method to mass clear cache data of all apps on my non-rooted Galaxy S8. It'd save a lot of space and was possible on ancient HTC phones with Android 2.3.5.
Edit: I experimented with wiping cache partition in recovery mode, but this ain't it - as the name suggests it's a separate partition.