Linguist John McWhorter has a fantastic course on human language on The Great Courses / Wondrium / Audible.

He specifically talks about this! Apparently the spoken words, the written word, song lyrics, and numbers are all processed in different parts of the brain almost as if they were different languages. He even mentions the same phenomenon as you and I think he also said this is the first part that gets hardwired into your brain as a kid, leaving you stuck counting in your mother tongue forever.