jessestcharles

One of the fantastic qualities of embedding based language models is that they provide a view on a semantic space that can be used quantitatively in most any downstream language task. As a conversational intelligence company Frame has many products that are enhanced by having a high quality domain specific language model to build on: tagging, sentiment, topic extraction, key words, summarization, etc. Best of all, these products can be iterated on in parallel! Improvements in a language model’s representation of a body of text should improve all downstream task without modification.

In this work we didn't explore classification performance characteristics. I suspect the nature of the misclassification at lower levels of domain data would revolve around the ways language usage differs in reviews vs common english. "Blockbuster" may have generally negative or neutral sentiment in a wikipedia based language model, perhaps most often referring to the failed rental chain. In the context of movie reviews "blockbuster" is almost always universally positive.

I think in this case the alignment between unlabeled domain data and the language task supports a convergence in language task performance. One argument to continue to prefer the ULM+domain model is that it is likely more generally capable if you remain in same domain but switch to a task that is less directly related to your unlabeled data. I haven’t seen any research that directly speaks to that intuition so it’s a good area for further study.

Yes, great insight! The choice to focus on sentiment here was mainly to align with fast.ai’s original research—-hopefully maximizing the generalizability and accessibility of the results.

Internally we have made use of these results to improve a broad set of language tasks, hopefully will be able to publish on those in the coming months as well.

Author here! Happy to answer any questions that folks have.