Re:A Gentle Introduction to Text Summarization in Machine Le...
by yorwba on Wednesday April 17, 2019 @04:22AM
(#19680722)
Not using spaces to separate words of course makes dealing with those languages a bit harder. The same problem applies to a lesser degree to languages like German, where compound words are fused together.
In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.
In those cases, the quality will depend heavily on the performance of the tokenizer as well. For Chinese you don't need a stemmer, since it's an analytic language, but Japanese is agglutinative and German is synthetic, so stemming is required for those.