What have I just said?
If the context is that it's been a handful of minutes, we don't say my usage is wrong; we definitely don't say that "sometimes days means minutes" and fret about how anyone will communicated time. We say that sometimes people exaggerate.
You can still object, if you wish, on stylistic grounds. You can object that you'd prefer we keep "literally" apart from some standard uses of words lest we allow inappropriate ambiguity. But none of that means anyone is using "literally" to mean "figuratively".
These are all literal meanings of insane. Of course, if we add figurative meanings we can increase the ambiguity further.
However, your criticism applies similarly to words like "truly" - if I say "that is truly insane", do I mean that it is insane in one of the literal senses of the word? Or the figurative uses? Am I just emphasizing either of these meanings, or do I feel a need to confirm that I am not lying?
Either way we take it, though, "literally" can never be replaced with "figuratively" without altering the meaning of a phrase. In it's use as an intensifier, it does NOT mean "figuratively", it means "very".
Also, looking on Merriam-Webster, they clearly discuss this and reach the same conclusion. They also mention that this meaning for emphasis appears as early as the 18th century, in the works of Charlotte Brontë, James Joyce, Mark Twain.
What have I just said?
If only we had a word whose express purpose was to avoid ambiguity for those times when it matters to communicate without ambiguity...