Re:The History and rationale of the Python 3 Unicode model f...
by ChrisSD on Wednesday December 14, 2022 @12:41PM
(#33986895)
Just to clarify further, note that actually preserving the "as if" behaviour does involve some complexity in the implementation. E.g. appending to WTF-8 has to be handled carefully to ensure it remains truly the same as doing so with WTF-16. This is because any newly paired surrogate has to be converted to its proper UTF-8 encoding. Similarly splitting WTF-8 can potentially break apart what was valid UTF-8 (though I'm not totally convinced that there's a good use case for actually doing this, at least for Windows paths).
Of course the implementation details are something that can and should be handled by a library instead of doing it manually.
Of course the implementation details are something that can and should be handled by a library instead of doing it manually.