News for hackers. Stuff that matters.
Re:Finnish breaks natural language processors by saagarjha on Friday March 01, 2019 @07:02AM (#19279751)
I have no stake in natural language processing, but it looks to me like a computer might be able to do a pretty good job at splitting that given a dictionary.
Re:Finnish breaks natural language processors by isani on Friday March 01, 2019 @07:15AM (#19279815)
Sure, you can get pretty far with a fairly simple solution. But lot of the time, you get two (or more) ways to split the string into dictionary words. For a simple English example, is it "justice was served" or "just ice was served"?
Re:Finnish breaks natural language processors by saagarjha on Friday March 01, 2019 @07:19AM (#19279851)
I guess that’s where context will have to be considered. Those two are valid sentences, so presumably humans are using context to distinguish between them, right?
Re:Finnish breaks natural language processors by MereInterest on Friday March 01, 2019 @08:25AM (#19280325)
The murderer came to my dinner party, and I had it all planned. In one of the ice cubes, I had frozen arsenic. The murderer would eat the same food, drink the same drink, and nobody would guess that they would die on leaving. When the evening was over, I knew what I would tell people.

Justicehadbeenserved.

Re:Finnish breaks natural language processors by woliveirajr on Friday March 01, 2019 @09:48AM (#19281077)
Please, share this with the world on tweeter.
Re:Finnish breaks natural language processors by MereInterest on Friday March 01, 2019 @10:30PM (#19287333)
If you would like to, feel free. For myself, I think that the comment's context of showing how ambiguity may not be resolved merely be contextual information is important, and that it would not stand as strongly without it.
Re:Finnish breaks natural language processors by plq on Friday March 01, 2019 @09:00AM (#19280633)
The stochastic strategy is to 1. enumerate every possible tag combination 2. assign a probability to each one 3. choose the parse with highest probability.

1. can be done either deterministically or stochastically.

2. requires you to have a language model trained with either human-tagged or semi-human-tagged corpus

3. was just the Viterbi algorithm last time I looked.

Implementing 1 and 2 are require broad domain knowledge in two very different domains (linguistics and machine learning respectively)

So while nowadays sentence segmentation can be considered a solved problem, it's far from trivial to implement one that can compete with the state of the art against real-world data.

There is also a nice body of deterministic (rule-based) literature that is practically ignored nowadays.