News for hackers. Stuff that matters.

A comment on Stanza: A Python natural language processing toolkit for many human languages.

Re:Stanza: A Python natural language processing toolkit for ... by dcsan on Tuesday March 24, 2020 @09:08PM (#22680940)
This does use spaCy for english tokenization, but has a bit wider support for languages in general.

eg spaCy has just finished their Chinese models in beta Discussion: https://github.com/howl-anderson/Chinese_models_for_SpaCy/is...

[ Parent | on HN ]
Re:Stanza: A Python natural language processing toolkit for ... by rpedela on Tuesday March 24, 2020 @09:49PM (#22681247)
You can use spacy for english tokenization or you can use their neural model. The neural model will generally do better, especially sentence segmentation, but will be slower.
[ Parent | on HN ]