Re:Advanced NLP with SpaCy
by wodenokoto on Thursday April 18, 2019 @05:31PM
(#19694987)
How good is spaCy for exploratory analysis of a corpus?
I'm thinking, questions like, top adjectives applied to men and women, or top word usage across document classes.
The best I've come across for doing this easily is in R, where they have tidytext[1] which is nice, and very straight forward to understand and work with. However, the data model stores each token and all its meta data (page number, sentence number, document id / title, document class, word type, etc, etc) in its own row, causing the in memory size of whatever corpus you are working on to explode.
I'm thinking, questions like, top adjectives applied to men and women, or top word usage across document classes.
The best I've come across for doing this easily is in R, where they have tidytext[1] which is nice, and very straight forward to understand and work with. However, the data model stores each token and all its meta data (page number, sentence number, document id / title, document class, word type, etc, etc) in its own row, causing the in memory size of whatever corpus you are working on to explode.
[1] https://www.tidytextmining.com