Re:Unicode sorting is hard and why browsers added special em...
by josephg on Monday June 28, 2021 @01:09AM
(#27658322)
Yeah its true but disappointing. I ported some email parsing code from C to wasm a couple of years ago and ran into this problem. Emails are encoded using all sorts of different character encodings. Canonicalizing them to utf-8 is important for consumers, but the libicu database for character encoding is several megabytes. So I'm left with a sad set of choices. I could:
1) Not support character encodings outside of ASCII, utf-8 and utf-16
2) Ship a huge wasm bundle or
3) Call back into javascript from my wasm module and use the javascript wrappers around libicu. This would make it harder to use the library outside of javascript.
I'm not sure what the right solution is here. If the character conversion databases can be shrunk to <500kb then I could just ship them with the module. Or if wasm had a better standard library, I could use that. But right now the whole situation is unfortunately awkward.
1) Not support character encodings outside of ASCII, utf-8 and utf-16
2) Ship a huge wasm bundle or
3) Call back into javascript from my wasm module and use the javascript wrappers around libicu. This would make it harder to use the library outside of javascript.
I'm not sure what the right solution is here. If the character conversion databases can be shrunk to <500kb then I could just ship them with the module. Or if wasm had a better standard library, I could use that. But right now the whole situation is unfortunately awkward.