News for hackers. Stuff that matters.

A comment on Japanese Grammar in EBNF notation.

Re:Japanese Grammar in EBNF notation by Grue3 on Sunday January 05, 2020 @04:27AM (#21960472)
My algorithm ( https://ichi.moe/ ) assumes almost nothing about the grammar and simply divides the sentences into the most valuable chunks (based on the word's length and how common it is), and then some sequences get bonus scores (for example noun followed by a particle). It performs surprisingly well in real life cases (where strict grammar is unlikely to be followed). I think it's the best algorithm that's currently available.

Example text from the blog post:

https://ichi.moe/cl/qr/?q=%E3%81%B2%E3%81%A8%E3%81%8C%E3%81%...

[ Parent | on HN ]
Re:Japanese Grammar in EBNF notation by GolDDranks on Sunday January 05, 2020 @06:33AM (#21960812)
What you are doing is essentially different: you are doing word sequencing and possibly part-of-speech tagging from a character stream. What is being done in this post, is parsing, which imposes a tree-like structure on the input sequence. These two techniques complement each other, and it might be beneficial to first do the word sequencing (the Viterbi algorithm is a good choice) and parsing after that.
[ Parent | on HN ]