Yes, recent work on transfer learning https://arxiv.org/abs/1801.06146 " Fine-tuned Language Models for Text Classification" suggests that training a lstm on a large corpora for language modeling, then using it as sentence encoder after tuning it in specific dataset gives a considerable perfomance.

This is covered in fast.ai 's new course 1.