Re:Unicode Normalization Forms: When ö ≠ ö
by smitty1e on Friday December 31, 2021 @07:41PM
(#29754166)
We had to ETL .csv data that must have originated in SQLServer.
The utf-16 fact about Windows was apparently unknown to my predecessor.
Who wrote some nasty c-language binary to copy the data, knock the upper byte off of each character ahead, and save the now ASCII text to a new file of the mysql load.
The encoding='utf-16' argument was all that was needed.
For want of a nail. . .
Re:Unicode Normalization Forms: When ö ≠ ö
by tialaramex on Friday December 31, 2021 @08:14PM
(#29754400)
I've had to fix this before. A co-worker working with data from a 3rd party supplier had gone "Oh this input data is mangled with stray zero bytes, I'll fix that" and of course that destroys any non-ASCII inputs, eventually I'm told that sometimes the import fails, I investigate, and I realise the "mangled" input is just UTF-16 encoded, conditionally remove the "strip zero bytes" hack and tell the decoder it's UTF-16 and it just works correctly.
The "maybe strip null bytes" code lived for years "just in case" after I fixed that because people couldn't believe that's all that was ever "wrong" with the data.
The perils of valuing backwards compatibility above all else… imagine having to use UTF16 in this day and age. Happy 2022!
Re:Unicode Normalization Forms: When ö ≠ ö
by vbezhenar on Saturday January 01, 2022 @12:00AM
(#29755670)
UTF-16 is a simple encoding. It should take a few dozens of LoC to convert to UTF-8. At least if you don’t need extreme performance with AVX, etc.
Re:Unicode Normalization Forms: When ö ≠ ö
by bjoli on Saturday January 01, 2022 @03:13AM
(#29756541)
My experience is exactly like what you say. Until it isn't. UTF16 seems like such a neat idea until it meets reality.
Mostly because of weird interactions of different libraries, languages and operating systems.
Re:Unicode Normalization Forms: When ö ≠ ö
by saati on Saturday January 01, 2022 @02:35AM
(#29756374)
> UTF-16 is a simple encoding
This isn't true, for starters it's two encodings UTF-16LE and UTF-16BE.
Re:Unicode Normalization Forms: When ö ≠ ö
by BlueTemplar on Friday December 31, 2021 @07:37PM
(#29754147)
Funnily base64 suffers from a related issue that the likes of base58 correct : l and I or O and 0 looking similar or even identical depending on the font !
Re:Unicode Normalization Forms: When ö ≠ ö
by dahfizz on Friday December 31, 2021 @08:07PM
(#29754355)
Why does that matter? When would a human need to read and comprehend base64 encoded data?
Re:Unicode Normalization Forms: When ö ≠ ö
by kadoban on Saturday January 01, 2022 @12:00AM
(#29755674)
Base58 is used for example for Bitcoin addresses. Being able to type an address from one system to another is a nice property, and it's much less error-prone if you don't have to worry about look-alike characters.
On *nix systems, I ran this through the base64 command; the data was UTF8, which meant that in practice it was ASCII (because we didn't have any special characters in our commit messages).
On Windows systems... oh god. The system treats all text as UTF-16 with whatever byte order, and it took me ages to figure out how to get it to convert the data to UTF-8 before encoding it. Eventually it started working, and it worked for a while until it didn't for whatever reason. I ended up tearing out all the code and just encoding the UTF-16 in base64 and then processing that into UTF-8 on the master where I had access to much saner tools.
Generally speaking, "Unicode" works great in most cases, but when you're dealing with systems with weird or unusual encoding habits, like Windows using UTF-16 or MySQL's "utf8" being limited to three bytes per unicode character instead of four, everything goes out the window and it's the wild west all over again.