Re:The dangers of streaming across versions of glibc: A caut...
by BeeOnRope on Tuesday January 08, 2019 @06:13PM
(#18860732)
So in summary it's a hard problem and requires some smarts to get right? Then we agree.
Retreating to ASCII is not the answer, and "full text search engines" don't even solve the same class of problem, such as say paginating a list of local towns by name.
Tools are not broken; reality is. So you have to work around assumptions that are pertinent to blind usage of technology.
You need to think how to use a tool (such as a Unicode-capable DB text type) to get a correct result. And first you need to understand what constitutes the correct result. Else you may end up with strings that are equally correct and nominally equal, but bytewise different, so search by index fails.