Re:cuneicode, and the Future of Text in C
by dark-star on Wednesday June 07, 2023 @01:52PM
(#36230551)
> There's no reason all the rest of your software needs to pay for Microsoft's mistake.
How is it a mistake when UTF was not even a thing when the NT kernel was developed?
Re:cuneicode, and the Future of Text in C
by tialaramex on Wednesday June 07, 2023 @02:16PM
(#36230975)
Not every mistake is something you could have prevented/ avoided. Sometimes you just get unlucky.
Windows APIs (including any COM and .NET libraries you may interoperate with) can serve you strings containing unpaired surrogates. That has nothing to do with that I “didn’t care about correctness”. Processing Windows UTF-16 as-is simply avoids complications that you would add by converting back and forth from and to UTF-8. If you’re working with other I/O (network requests, file contents) that is UTF-8-based, it makes much more sense to strategically pick the places where you perform conversions within your application. The conversion then is also limited to where data actually crosses between Windows APIs and other I/O within your application.
Traditionally, strings in C are encoding-agnostic. By default, the encoding of the current locale is assumed, and applications are expected to convert to other encodings when specifically needed. Forcing everything into “there can only be one encoding we can work with” isn’t a great solution. One can wish that everything was UTF-8 from the beginning, but that’s not the world we live in.
There's no reason all the rest of your software needs to pay for Microsoft's mistake.
> and converting to/from UTF-8 on every Windows API call is not only a pessimization both in runtime and memory usage, but also introduces complications in having to handle possible conversion failures (involving unpaired surrogate characters, in particular).
If the text isn't actually Unicode, then all you're demonstrating by not handling that in another language is that you didn't care about correctness. There's no magic here, unpaired surrogates aren't somehow actually valid Unicode so long as you stick with C.