News for hackers. Stuff that matters.
Re:Wink-lemmatizer by afsina on Sunday June 24, 2018 @07:59AM (#17386200)
Good luck with Turkish. A very large lookup table may work for %90 cases but it will still fail. A basic `true` lemmatizer requires either a complex graph with rules, or an FST generated from it. input is searched through the graph and morphological disambiguator must be applied to result to pick the correct lemma.
Re:Wink-lemmatizer by DFHippie on Sunday June 24, 2018 @09:00AM (#17386382)
Or Welsh. You want a canonical form for "wnaethpwyd"? Try "gwneud"! Some regexes and an exceptions list isn't going to cut it.

The more a language needs a lemmatizer for NLP the harder it is to write it.

Re:Wink-lemmatizer by tomjakubowski on Sunday June 24, 2018 @11:51AM (#17386971)
Could someone explain what's hard in particular about canonical forms in Welsh?