News for hackers. Stuff that matters.
Re:A Gentle Introduction to Text Summarization in Machine Le... by rpedela on Tuesday April 16, 2019 @05:35PM (#19677547)
Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work.

For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.

There are no comments on this comment yet.