Re:Using sqlite3 as a notekeeping document graph
by laurent123456 on Sunday July 04, 2021 @04:12AM
(#27728059)
I believe there are tokenizers for Chinese language but haven't tried them since they aren't available by default in Android or iOS for example. In our app we've ended up having two modes - one for alphabetical languages and one for Thai, Chinese, Hebrew and Arabic, but in this case it's not actually using FTS but a plain SQL query.
Other than that, there's also tokenizing (splitting text into words) that's also unicode defined and stemming (reducing tokens to a base stem like "likes"->"lik-" in English)