To add to that, some languages use some of the same characters from the English language, but formatted differently in order to fit into that language's rules. For example, Japanese characters are the same width, so in Unicode there are the number and letters with full width in order to fit in when English is mixed in with Japanese.
The practical part of me agrees with patio11, and that the knowledge gain of having these semantics inherent in the file is offset many times by possibly having to treat different bytes as the same character semantically.