There is no silver bullet. The best solution is to collect more data to bring the classes to a balance. The second best approach is to try algorithms like SMOTE.
To clarify: fasttext does not support n-grams of words, but instead considers n-grams of characters within words.
See options -minn, -maxn and -wordNgrams.
For the FastText model, it would be whatever FastText is doing under the hood. I haven't peeked at the code to see what it is doing, but Section 2 of the paper I cited above seems to imply that an average is taken.
[0] https://github.com/keras-team/keras/blob/master/examples/imd...
As with anything, your mileage may vary.
One aspect of FastText that definitely helped in my case was n-gram support (both word and character, tunable via command-line arguments). In my corpus, I have short fragments of sentences containing misspelled words, incorrect grammar etc. plus my test set has out-of-vocabulary words.
n-grams are more robust to these than Word2Vec which uses a static vocabulary.
[0] http://www.aclweb.org/anthology/P/P08/P08-1028
Other approaches like doc2vec are also used.
"Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used."
I thought the approach you described was doc2vec. If not, then does it have a name/citation?
Either way... Very nice, thanks!
For my problem (1000+ total classes, 1 class per input), I experimented with Naive Bayes + TFIDF (~50% accuracy, < 1 sec training), then Word2vec + CNN model on GPU (~70% accuracy, 6 hrs training), and finally FastText (99% accuracy, 10 minutes training).
FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams, and there is a default implementation in C++ with a built-in classifier that runs on the command-line (no need to setup Tensorflow or anything like that).
Despite it only running on plain CPUs and only supporting a linear classifier, it seems to beat GPU-trained Word2Vec CNN models in both accuracy and speed in my use cases. I later discovered this paper from the authors comparing CNNs (and other algorithms) to FastText, and their results track my experiences [1].
This goes to show that while GPU-accelerated models are cool, sometimes using a simpler, more suitable model can have a significantly better pay-off.
[0] https://fasttext.cc/docs/en/supervised-tutorial.html
[1] https://arxiv.org/abs/1607.01759