There's no such thing as communication without ambiguity while using natural language. In your particular example, any interpretation depends crucially on what "that" might be referring to. It could refer to an animal, in which case you may mean that it seems to be suffering from a mental illness (maybe it has rabies) OR that it is unable to think clearly (it is insane with hunger, or excitement). It could be referring to an action, which may mean that it is either the action of someone suffering from a mental illness, or the action of someone being temporarily unable to think clearly, or it is an absurd action.

These are all literal meanings of insane. Of course, if we add figurative meanings we can increase the ambiguity further.

However, your criticism applies similarly to words like "truly" - if I say "that is truly insane", do I mean that it is insane in one of the literal senses of the word? Or the figurative uses? Am I just emphasizing either of these meanings, or do I feel a need to confirm that I am not lying?

Either way we take it, though, "literally" can never be replaced with "figuratively" without altering the meaning of a phrase. In it's use as an intensifier, it does NOT mean "figuratively", it means "very".

Also, looking on Merriam-Webster, they clearly discuss this and reach the same conclusion. They also mention that this meaning for emphasis appears as early as the 18th century, in the works of Charlotte Brontë, James Joyce, Mark Twain.