UTF-8 Everywhere (2012)

Re:UTF-8 Everywhere (2012) by Avernar on Monday June 20, 2016 @04:39AM (#11936824)

State machines would have to operate on the byte level. Otherwise each state would have to have have 65536 entries per state. The trick to handle UTF-8 would be to have 0-127 run as a state machine and > 127 break out to functions to handle the various unicode ranges that are valid for identifiers.

For languages that only allow non ascii in string literals a pure state machine would suffice.

Not sure why you're mentioning parsers. At that point you you're dealing with tokens.

As for UTF-16 it's an ugly hack that never should have existed in the first place. Unfortunately the unicode people had to fix their UCS-2 mistake.

Since Javascript is standardised to be either UCS-2 or UTF-16 it probably made sense to make the scanner use UTF-16.

[ Parent | on HN ]

Re:UTF-8 Everywhere (2012) by dietrichepp on Wednesday June 22, 2016 @10:13AM (#11954071)

State machines don't have to operate on the byte level because the tables can use equivalency classes. This will often result in smaller and faster state machines than byte-level state machines, if your language uses Unicode character classes here and there.

[ Parent | on HN ]