Re:Libgrapheme: A simple freestanding C99 library for Unicode
by hnfong on Wednesday November 16, 2022 @12:55PM
(#33627468)
The claims about "word segmentation" caught my eye, because it's not even well defined in my native language. (one example among the difficulties: there's a lot of constructs like "rid-fucking-iculous". Should it be "rid/fucking/iculous" or "fucking ridiculous"?)
After checking the source to see how they perform this literally impossible feat, it turns out they were implementing a Unicode standard that basically tries to do something useful (for some families of languages) but has a whole wall of caveats: https://unicode.org/reports/tr29/#Word_Boundary_Rules
I can see users of the word segmentation function shooting their feet really badly with at least CJK languages, especially with a deceptively simple API like that. In general "word segmentation" doesn't make sense on a language-neutral level. libgrapheme even admits to this somewhat:
""" For some languages, for instance, it is necessary to have a dictionary on hand to always accurately determine when a word begins and ends. The defaults provided by the standard, though, already do a great job respecting the language's boundaries in the general case """
I disagree about the "great job" part. It's probably the best job a Unicode spec can do, but it's not going to be great for like at least 20% of the world's population... (Also a minor nit is that no dictionary is accurate, so even with a dictionary the results are not really "always accurate", just mostly accurate, unless you hit edge cases [for CJK at least], upon which advanced NLP techniques are needed.)
IMHO they should have a big disclaimer in their GRAPHEME_NEXT_WORD_BREAK(3) manual page warning users about the caveats (like, "if you're using this for anything other than $(these families of languages) make sure you really know what you're doing before using this function").
After checking the source to see how they perform this literally impossible feat, it turns out they were implementing a Unicode standard that basically tries to do something useful (for some families of languages) but has a whole wall of caveats: https://unicode.org/reports/tr29/#Word_Boundary_Rules
I can see users of the word segmentation function shooting their feet really badly with at least CJK languages, especially with a deceptively simple API like that. In general "word segmentation" doesn't make sense on a language-neutral level. libgrapheme even admits to this somewhat:
""" For some languages, for instance, it is necessary to have a dictionary on hand to always accurately determine when a word begins and ends. The defaults provided by the standard, though, already do a great job respecting the language's boundaries in the general case """
I disagree about the "great job" part. It's probably the best job a Unicode spec can do, but it's not going to be great for like at least 20% of the world's population... (Also a minor nit is that no dictionary is accurate, so even with a dictionary the results are not really "always accurate", just mostly accurate, unless you hit edge cases [for CJK at least], upon which advanced NLP techniques are needed.)
IMHO they should have a big disclaimer in their GRAPHEME_NEXT_WORD_BREAK(3) manual page warning users about the caveats (like, "if you're using this for anything other than $(these families of languages) make sure you really know what you're doing before using this function").