Nope, big mistake. You cannot just search for "Café" in "Café", as there are multiple representations for the same character. Think of Cyrillic vs Greek, Han vs Hangul. The popular garbage-in, garbage-out doesn't apply to unicode identifiers. That's why the Mac filesystem normalization was correct, and the unix filesystems are all broken. If it's not identifiable it's no identifier. That's why java and cperl are the only languages with proper and safe unicode support. Python 3 is a bit better than Python 2, Perl6 better than Perl5 and the rest, but still don't apply unicode security guidelines for identifiers. It's much better for browsers and email clients.

And with case-insensitivity it gets worse, as there are some locale dependent additional rules, for Turkish and Lithuania. And this depends on "some" global settings.

http://perl11.org/blog/foldcase.html

Japanese is also great. While there are general rules when you use hiragana vs. katakana, sometimes for stylistic reasons the "wrong" one is used, or some kanji is "spelled out" as hiragana, and depending on context, you might or might not want to include these in your search results.

Tbh I just stopped caring and now happily live in my little world where utf8 is the solution to everything, doesn't yield any problems ever, and anyone telling me differently gets completely ignored.

If you're using WinAPI, you have to use UCS-16. Converting strings for every call would be abysmal to performance.

For most software it's perfectly fine to convert. What Windows API call that takes string input do you call so often that it would be an issue?

There are only very few WinAPI functions which use UCS-2 vs ASCII, and none of them are performance-critical. It's totally fine to convert from/to UTF-8 on the fly, especially since the conversion is extremely cheap (and the rest of the world has moved on to UTF-8 anyway).

All winapi functions use UCS2. The "ASCII" versions are just wrappers that implicitly convert from the current locale (latin1, ShiftJIS, BIG5, ...) to UCS2. Try passing some Japanese text to the ASCII version of a function on a system that's set to eg. German and see what happens...

Most Win32 API functions don't even take string parameters, so there's no separate ASCII or UCS-2 version of those.

Of the functions that take strings, the old ...A() versions should never be called anyway since they depend on the obsolete concept of code pages. The proper way to work with strings on Windows (IMHO) is to encode all string data to UTF-8, only encode/decode from/to UCS-2 when needed, and call the ...W() functions explicitely (don't use the global UNICODE define).

Also see: http://utf8everywhere.org/