It's not so simple: it depends on what you're doing with the text. If you're not trying to do analysis with it, encoded text is more or less a program written in a DSL that, when interpreted by a font renderer, draws symbols in some graphical context. Depending on the analysis you want to do, you need varying amounts of knowledge. Perhaps you only need to know about word boundaries; perhaps you're trying to look things up in a normalized dictionary; maybe even decompose a word into phonemes to try and pronounce it. These require different levels of analysis, and one size won't fit all.