Re:How the JVM compares strings on x86 using pcmpestri
by banthar on Sunday January 07, 2018 @02:33PM
(#16092201)
Even outside Web, you still have mostly-ASCII:
* filenames
* identifiers
* config files
* text protocols
* host names, email addresses
* embedded scripts (including SQL and OpenGL shaders)
* command line interfaces
* translations for languages using Latin alphabets
I don't think 2/3 size reduction for some languages will offset the cost in all the other places.
Re:How the JVM compares strings on x86 using pcmpestri
by Const-me on Monday January 08, 2018 @10:01AM
(#16097130)
For some of these things we don’t have much choice, because the encoding is part of some lower-level API (file system, OpenGL, CLI), which usually don’t accept arbitrary encoding. They accept only one, and unless you want to waste time converting, you better use that exact encoding.
Other stuff like IDs, shaders before GL 4.2, and many text protocols aren’t Unicode at all.
For configs I usually use UTF-8 myself, because I don’t like writing parsers for custom formats and just use XML, and any standard-compliant parser supports all of them.
I don’t advocate using UTF16 for the web, but people still code native desktop apps, mobile apps, embedded software, videogames, store stuff in various databases, etc. For such use, markup is irrelevant.