You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
Yes. String processing often occurs in a loop (like for trim, contains, split ...).
You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into.
Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.
I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now.
And Unicode has a non-negligible footprint.
I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU).
Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ...
I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.