News for hackers. Stuff that matters.
Re:How the JVM compares strings on x86 using pcmpestri by Const-me on Sunday January 07, 2018 @12:25PM (#16091625)
> most common operations are comparisons for equality and copying anyways

Indexing & substrings are common, too.

> These days code is so often bandwidth limited if anything

Right, and for 1 billion Chinese speaking people UTF16 is 2 bytes/character, UTF8 is 3 bytes/character.

Re:How the JVM compares strings on x86 using pcmpestri by vardump on Sunday January 07, 2018 @12:40PM (#16091707)
> Indexing & substrings are common, too.

Indexing code points in both UTF-8 and UTF-16 requires reading the whole string up to index location. Substrings are the same as well.

> Right, and for 1 billion Chinese speaking people UTF16 is 2 bytes/character, UTF8 is 3 bytes/character.

That's true for a text file without markup. But most text is not like that in 2017. HTML is probably the most common text format nowadays.

So let's see how a popular Chinese language website does.

  curl http://language.chinadaily.com.cn/ --silent | wc -c
  52678

  curl http://language.chinadaily.com.cn/ --silent | iconv -f utf8 -t utf-16le | wc -c
  93368
So UTF-8 seems to be quite a bit more efficient in this case, 52678 bytes. When converted to UTF-16, same page was 93368 bytes.
Re:How the JVM compares strings on x86 using pcmpestri by Ded7xSEoPKYNsDd on Sunday January 07, 2018 @01:10PM (#16091838)
> Indexing code points in both UTF-8 and UTF-16 requires reading the whole string up to index location. Substrings are the same as well.

Java's String functions don't index by Unicode code points, though. Java strings are encoded in UCS-2, or at least the API needs to pretend that they are.

Re:How the JVM compares strings on x86 using pcmpestri by vardump on Sunday January 07, 2018 @04:27PM (#16092843)
That can sure lead to interesting bugs!
Re:How the JVM compares strings on x86 using pcmpestri by Const-me on Sunday January 07, 2018 @01:21PM (#16091880)
Right. Same in C#, C++ STL, and in Apple’s obj-c/swift.
Re:How the JVM compares strings on x86 using pcmpestri by tjalfi on Sunday January 07, 2018 @02:37PM (#16092221)
Swift takes a different approach than Objective C[0].

[0] https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

Re:How the JVM compares strings on x86 using pcmpestri by Const-me on Sunday January 07, 2018 @12:54PM (#16091763)
Even in 2017, not everyone is a Web or Electron developer. I certainly am not.

I don’t advocate using UTF16 for the web, but people still code native desktop apps, mobile apps, embedded software, videogames, store stuff in various databases, etc. For such use, markup is irrelevant.

Re:How the JVM compares strings on x86 using pcmpestri by banthar on Sunday January 07, 2018 @02:33PM (#16092201)
Even outside Web, you still have mostly-ASCII:

* filenames

* identifiers

* config files

* text protocols

* host names, email addresses

* embedded scripts (including SQL and OpenGL shaders)

* command line interfaces

* translations for languages using Latin alphabets

I don't think 2/3 size reduction for some languages will offset the cost in all the other places.

Re:How the JVM compares strings on x86 using pcmpestri by Const-me on Monday January 08, 2018 @10:01AM (#16097130)
For some of these things we don’t have much choice, because the encoding is part of some lower-level API (file system, OpenGL, CLI), which usually don’t accept arbitrary encoding. They accept only one, and unless you want to waste time converting, you better use that exact encoding.

Other stuff like IDs, shaders before GL 4.2, and many text protocols aren’t Unicode at all.

For configs I usually use UTF-8 myself, because I don’t like writing parsers for custom formats and just use XML, and any standard-compliant parser supports all of them.

Re:How the JVM compares strings on x86 using pcmpestri by pjmlp on Monday January 08, 2018 @03:15AM (#16095367)
If English is your world yeah.

Some of us use other languages and like to use them everywhere we can.

Re:How the JVM compares strings on x86 using pcmpestri by jcranmer on Sunday January 07, 2018 @12:37PM (#16091687)
Represent indexes not as number-of-code-points from start but as byte offsets, and index/substring doesn't need to decode code points. You lose the ability to easily say "get me the 100th code point in this string", but I'm hard-pressed to actually think of any case where that is actually valuable.

> Right, and for 1 billion Chinese speaking people UTF16 is 2 bytes/character, UTF8 is 3 bytes/character.

The information density of a single hanzi character is roughly equivalent to 5 letters in English. A Chinese plaintext document in UTF-8 is still smaller in memory footprint than an equivalent English document in ASCII. Of course, most documents aren't plaintext, and where people use characters for metadata (e.g., email, HTML), there is a substantial corpus of ASCII metadata in those documents that UTF-8 is still smaller than UTF-16 even for East Asian languages.

Of course, it's moot since the people who don't like UTF-8 in China and Japan aren't using UTF-16 either. They're using GB18030 or ISO-2022-JP for their documents.

Re:How the JVM compares strings on x86 using pcmpestri by Const-me on Sunday January 07, 2018 @12:50PM (#16091744)
> A Chinese plaintext document in UTF-8 is still smaller in memory footprint than an equivalent English document in ASCII.

When you need to process Chinese text you don’t care how much an equivalent English document would take. You only care about the difference between different encodings of Chinese language. And UTF16 is more compact for East Asian languages.

> most documents aren't plaintext

That’s true for the web, and that’s why UTF8 is the clear winner there. In a desktop software, in a videogame, in a database — not so much.