> an extra layer of depth

I think that's the best way to put it. Like you intuited, a phonetic system is perfectly capable of writing down spoken conversations, but without as much context as a spoken conversation it's going to get more confusing. For example, he she and it in Chinese are all pronounced "tā", but are written as 他，她， and 它 respectively in Chinese. In fact it's common to write "ta" as an equivalent to xhe/xer/xem.