News for hackers. Stuff that matters.
Re:Wink-lemmatizer by slx26 on Sunday June 24, 2018 @04:28AM (#17385594)
any plans for multi-language support?
Re:Wink-lemmatizer by EmilStenstrom on Sunday June 24, 2018 @07:51AM (#17386170)
This is really needed in the NLP world. Most things are english only.
Re:Wink-lemmatizer by afsina on Sunday June 24, 2018 @07:59AM (#17386200)
Good luck with Turkish. A very large lookup table may work for %90 cases but it will still fail. A basic `true` lemmatizer requires either a complex graph with rules, or an FST generated from it. input is searched through the graph and morphological disambiguator must be applied to result to pick the correct lemma.
Re:Wink-lemmatizer by DFHippie on Sunday June 24, 2018 @09:00AM (#17386382)
Or Welsh. You want a canonical form for "wnaethpwyd"? Try "gwneud"! Some regexes and an exceptions list isn't going to cut it.

The more a language needs a lemmatizer for NLP the harder it is to write it.

Re:Wink-lemmatizer by tomjakubowski on Sunday June 24, 2018 @11:51AM (#17386971)
Could someone explain what's hard in particular about canonical forms in Welsh?
Re:Wink-lemmatizer by wyldfire on Sunday June 24, 2018 @09:35AM (#17386495)
spaCy excels here. English, German, Portuguese, and more.
Re:Wink-lemmatizer by slx26 on Sunday June 24, 2018 @11:58AM (#17386991)
I can personally suggest FreeLing [nlp.lsi.upc.edu/freeling]. It supports most languages in the iberian peninsula (spanish, portuguese, catalan (including multiple dialects), galician), and then it also includes english, italian, french, german, russian, croatian and slovene. It's also more flexible than other tools if you need to go outside those languages and customize something.
Re:Wink-lemmatizer by sanjayaksaxena1 on Monday June 25, 2018 @11:11AM (#17393005)
As of now wink-tokenizer (https://github.com/winkjs/wink-tokenizer) supports multiple scripts, therefore, it can also tokenize sentences in languages like Hindi, Marathi, French, German etc. We are working on extending multi-lingual support to other components including this lemmatizer.