Re:Why to normalize Unicode strings
by magduf on Wednesday March 13, 2019 @01:16PM
(#19380952)
Sorry if this sounds a little clueless, but it sounds like the problem is because there's multiple Unicode standards (UTF-8, UTF-16, UTF-32). So it seems like if you just re-encode everything into one of these before committing to the storage layer, that you'd avoid this problem altogether, and you'd be able to do operations in the storage layer correctly too.
[ Parent
| on HN ]
Re:Why to normalize Unicode strings
by happytoexplain on Wednesday March 13, 2019 @01:57PM
(#19381485)
A Unicode string is an array of Unicode's building blocks: Code points. Normalization/composition refers to how we use code points to represent characters. Encoding refers to how we use bytes to represent code points. The two concepts exist at two independent levels of abstraction.
Re:Why to normalize Unicode strings
by dummyfunnytoo on Wednesday March 13, 2019 @04:12PM
(#19383244)
Accurate but so arrogant. HN purity achieved.
Re:Why to normalize Unicode strings
by happytoexplain on Wednesday March 13, 2019 @04:17PM
(#19383293)
Sorry, I didn't intend that. I honestly considered it a reasonable question.
No worries, your reply was a concise clarification of the issue and terminology, and a reasonable answer; I didn't perceive it as arrogant in the least.
OP here; I didn't perceive it as arrogant either. I think the "arrogant" accuser might have a point about some stuff I see on HN, but this reply to my post just isn't it.
Re:Why to normalize Unicode strings
by jrochkind1 on Wednesday March 13, 2019 @01:18PM
(#19380985)
Yeah, no. That is actually a different issue. Curious if you read the OP?
But, no, you do not avoid issues with unicode normalization by ensuring everything is in (eg) UTF-8. Unicode normalization is a thing within UTF-8.