News for hackers. Stuff that matters.
Re:Finnish breaks natural language processors by plq on Friday March 01, 2019 @05:04AM (#19279207)
These are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research.

While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology.

Also, a small terminology fix: Verbs are "conjugated" but names/objects are "declinated".

[1]: https://github.com/ahmetaa/zemberek-nlp

Re:Finnish breaks natural language processors by tallanvor on Friday March 01, 2019 @06:50AM (#19279700)
But a spell checker is just a tiny part of the NLP domain.

When people talk about NLP, they are usually referring to understanding the meaning of the entire text being analyzed, rather than just providing meanings for individual words. A spell checker can't tell you the intent or subtleties of what is being conveyed.

Re:Finnish breaks natural language processors by plq on Friday March 01, 2019 @08:50AM (#19280558)
> But a spell checker is just a tiny part of the NLP domain.

Sure, but the linked tweet claims that not even a working spellchecker exists for Finnish. I'm sure Joakim Nivre would think otherwise !

Re:Finnish breaks natural language processors by szatkus on Friday March 01, 2019 @05:23AM (#19279298)
What's the problem with Japanese? It's highly regular language, so it should be easy to tokenize.

AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.

Re:Finnish breaks natural language processors by Grue3 on Friday March 01, 2019 @10:27AM (#19281445)
You can do it for dictionary words with a relatively high degree of accuracy, but it's very hard. I have developed the most accurate tokenizer that currently exists [1], but there are still sentences where it fails. Words written in hiragana create an inherent ambiguity because almost any combination of hiragana can be split into existing words in several ways.

Another problem is proper names. This one is inherently unsolvable, because anything can be a word. You can have a dictionary of proper names, but there will still be people, companies, fictional characters who aren't in the dictionary.

[1] https://github.com/tshatrov/ichiran

Re:Finnish breaks natural language processors by yorwba on Friday March 01, 2019 @11:52AM (#19282580)
How do you test the accuracy of your tokenizer? Is there a database of ground-truth tokenized sentences that you evaluate on? It would be nice to see some kind of benchmark results comparing with MeCab.
Re:Finnish breaks natural language processors by Grue3 on Friday March 01, 2019 @12:25PM (#19283070)
It's not a matter of benchmarks really. It's more of a matter of MeCab simply not knowing that certain grammar exists. For example 〜ちゃった for 〜てしまった is a pretty common form. MeCab doesn't understand the word 忘れちゃった because it only knows 忘れて and しまった as completely separate words and can't combine them in any way, but Ichiran can because my word-form generator is much more sophisticated. This is also why I cannot compare the results of segmentation directly: I combine words and their suffixes into a single "compound word" which has a tree-like recursive structure [1]. MeCab and other tokenizers can only detect basic forms and leave suffixes as a completely separate "word". Which makes them automatically inferior.

[1] https://ichi.moe/cl/qr/?q=%E5%BF%98%E3%82%8C%E3%81%A1%E3%82%...

Re:Finnish breaks natural language processors by yorwba on Friday March 01, 2019 @06:04PM (#19286174)
If I do echo "忘れちゃった" | mecab, I get

  忘れちゃ 動詞,*,母音動詞,タ系連用チャ形,忘れる,わすれちゃ,代表表記:忘れる/わすれる 付属動詞候補(基本) 反義:動詞:覚える/おぼえる
  った 接尾辞,動詞性接尾辞,子音動詞ワ行,タ形,う,った,代表表記:う/う
  EOS
So MeCab most definitely knows about that grammar. It's special-cased even! Admittedly, the way った is split off as a separate suffix is a bit ugly.
Re:Finnish breaks natural language processors by Grue3 on Saturday March 02, 2019 @03:07AM (#19288203)
That's the thing though, 忘れちゃ is a completely different, unrelated form that means 忘れて+は. So yes, this split is in fact incorrect and if it wasn't past tense, but dictionary form 忘れちゃう, the result would be even worse with う being split off as "rain"? or something.
Re:Finnish breaks natural language processors by isani on Friday March 01, 2019 @06:44AM (#19279679)
Japanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous.

Englishwouldbemuchhardertoparseifwrittenlikethis.

Re:Finnish breaks natural language processors by saagarjha on Friday March 01, 2019 @07:02AM (#19279751)
I have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.
Re:Finnish breaks natural language processors by isani on Friday March 01, 2019 @07:15AM (#19279815)
Sure, you can get pretty far with a fairly simple solution. But lot of the time, you get two (or more) ways to split the string into dictionary words. For a simple English example, is it "justice was served" or "just ice was served"?
Re:Finnish breaks natural language processors by saagarjha on Friday March 01, 2019 @07:19AM (#19279851)
I guess that’s where context will have to be considered. Those two are valid sentences, so presumably humans are using context to distinguish between them, right?
Re:Finnish breaks natural language processors by MereInterest on Friday March 01, 2019 @08:25AM (#19280325)
The murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people.

Justicehadbeenserved.

Re:Finnish breaks natural language processors by woliveirajr on Friday March 01, 2019 @09:48AM (#19281077)
Please, share this with the world on tweeter.
Re:Finnish breaks natural language processors by MereInterest on Friday March 01, 2019 @10:30PM (#19287333)
If you would like to, feel free. For myself, I think that the comment's context of showing how ambiguity may not be resolved merely be contextual information is important, and that it would not stand as strongly without it.
Re:Finnish breaks natural language processors by plq on Friday March 01, 2019 @09:00AM (#19280633)
The stochastic strategy is to 1. enumerate every possible tag combination 2. assign a probability to each one 3. choose the parse with highest probability.

1. can be done either deterministically or stochastically.

2. requires you to have a language model trained with either human-tagged or semi-human-tagged corpus

3. was just the Viterbi algorithm last time I looked.

Implementing 1 and 2 are require broad domain knowledge in two very different domains (linguistics and machine learning respectively)

So while nowadays sentence segmentation can be considered a solved problem, it's far from trivial to implement one that can compete with the state of the art against real-world data.

There is also a nice body of deterministic (rule-based) literature that is practically ignored nowadays.

Re:Finnish breaks natural language processors by gizmo686 on Friday March 01, 2019 @08:59AM (#19280625)
But Japanese is not written as character soup. It mixes two (actually 3) types of characters, with the "grammatical" sounds being written in hiragana and most content sounds being written in kanji. Since the grammatical sounds are a closed class, and tend to occur at word boundries, it turns out to be relativly simple to seperate words.
Re:Finnish breaks natural language processors by szatkus on Friday March 01, 2019 @07:20AM (#19279853)
Isn't it a case when parsing other languages from speech? Are there any audible cues between words when we speak?
Re:Finnish breaks natural language processors by fluffycat on Friday March 01, 2019 @06:06AM (#19279496)
Turkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity.

I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.

Re:Finnish breaks natural language processors by szatkus on Friday March 01, 2019 @06:35AM (#19279636)
In Japanese I think it's pretty straightforward to trace the original word. But there is a fair share of homophones, so if your input is speech-to-text you already have many possibilities without much grammar.