News for hackers. Stuff that matters.
Re:cuneicode, and the Future of Text in C by layer8 on Wednesday June 07, 2023 @02:14PM (#36230948)
Windows APIs (including any COM and .NET libraries you may interoperate with) can serve you strings containing unpaired surrogates. That has nothing to do with that I “didn’t care about correctness”. Processing Windows UTF-16 as-is simply avoids complications that you would add by converting back and forth from and to UTF-8. If you’re working with other I/O (network requests, file contents) that is UTF-8-based, it makes much more sense to strategically pick the places where you perform conversions within your application. The conversion then is also limited to where data actually crosses between Windows APIs and other I/O within your application.

Traditionally, strings in C are encoding-agnostic. By default, the encoding of the current locale is assumed, and applications are expected to convert to other encodings when specifically needed. Forcing everything into “there can only be one encoding we can work with” isn’t a great solution. One can wish that everything was UTF-8 from the beginning, but that’s not the world we live in.

There are no comments on this comment yet.