What about compiling programs with `cl -utf-8` and using -A APIs? I know the docs say “If the ANSI code page is configured for UTF-8, -A APIs typically operate in UTF-8”, to which I say, just “typically”?! But in practice is it not sufficient?

The -A APIs are converted to calls to the -W APIs, IIRC, so you pay that cost everywhere. Not sure if that's universally true anymore, but it has been that way at least on the NT based kernel since...inception? Probably a good question for Raymond Chen. Sadly, I can't seem to find anything relevant in a quick search of the Old New Thing.

The GP is referencing the new UTF-8 APIs that reuse the -A suffixes.[0]

> Until recently, Windows has emphasized "Unicode" -W variants over -A APIs. However, recent releases have used the ANSI code page and -A APIs as a means to introduce UTF-8 support to apps. If the ANSI code page is configured for UTF-8, -A APIs typically operate in UTF-8. This model has the benefit of supporting existing code built with -A APIs without any code changes.

This requires Windows 10 version 1903 or newer. As the version number suggests, this feature is only four years old, and using them prevents your program from working on older versions.

[0]: https://learn.microsoft.com/en-us/windows/apps/design/global...

Yes, precisely. I have never tested it conclusively, but (taking it as a given that we are targeting >=1903) is it not sufficient and optimal to use `-utf-8` and -A APIs?

Windows UTF-8 support is relatively recent and I have no experience with it, so I don’t know. However I expect Windows to just do the same conversions internally or in the linked runtime, that the program would otherwise have to do by itself. I’d assume that there will be edge cases that such programs then can’t handle, such as UI input and file paths containing unpaired surrogates.

UTF-8 should never contain surrogates as they serve no purpose when all code points can be encoded directly. That doesn't prevent programmers doing stupid things with them but that's not a Unicode problem.

Correct – but that’s exactly why you can’t just say “all strings are UTF-8, we’re done here”. Because Windows filenames can contain unpaired surrogates, which don’t have a sensible UTF-8 translation. (Unix has a similar problem with filenames being arbitrary bytes that may not be valid UTF-8.)

Note that Rust has OsString to handle both of these cases, but it’s somewhat hard to use.

With networking code you sometimes need an OsString for the platform you're not on. Rust doesn't seem to support this, other than by writing your own.

> the native encoding is UTF-16

There's no reason all the rest of your software needs to pay for Microsoft's mistake.

> and converting to/from UTF-8 on every Windows API call is not only a pessimization both in runtime and memory usage, but also introduces complications in having to handle possible conversion failures (involving unpaired surrogate characters, in particular).

If the text isn't actually Unicode, then all you're demonstrating by not handling that in another language is that you didn't care about correctness. There's no magic here, unpaired surrogates aren't somehow actually valid Unicode so long as you stick with C.

> There's no reason all the rest of your software needs to pay for Microsoft's mistake.

How is it a mistake when UTF was not even a thing when the NT kernel was developed?

Not every mistake is something you could have prevented/ avoided. Sometimes you just get unlucky.

Windows APIs (including any COM and .NET libraries you may interoperate with) can serve you strings containing unpaired surrogates. That has nothing to do with that I “didn’t care about correctness”. Processing Windows UTF-16 as-is simply avoids complications that you would add by converting back and forth from and to UTF-8. If you’re working with other I/O (network requests, file contents) that is UTF-8-based, it makes much more sense to strategically pick the places where you perform conversions within your application. The conversion then is also limited to where data actually crosses between Windows APIs and other I/O within your application.

Traditionally, strings in C are encoding-agnostic. By default, the encoding of the current locale is assumed, and applications are expected to convert to other encodings when specifically needed. Forcing everything into “there can only be one encoding we can work with” isn’t a great solution. One can wish that everything was UTF-8 from the beginning, but that’s not the world we live in.

Rust has an abstraction for the system encoding: OsStr. Though I think most Rust programs just assert everything is UTF-8, the language goes to great lengths to ensure it is possible, and not even particularly difficult, to write Rust code that can handle Windows's broken UTF-16, yet which isn't Windows-specific.

Eh, the `OsStr` API has warts still, some parts are unstable (https://github.com/rust-lang/rust/issues/111544).