We used a technique similar to LSA.
Our first step was to build a bag of words and construct a scaled TF off of that. Then we verified the label for about 10% of the data and used that as our training set. Using cosine similarity (which we calculated using matrix multiplication of tfs) we found top n labeled documents that were similar to document in question to decide the label of remaining 90% documents.
Once we had this dataset we ran it against logistic regression as well by training on same 10% and use remaining to find the label. Interestingly document similarity was only slightly better than logistic regression. Logistic was 10 times faster.
I think this approach worked for us because we had somewhat of a mutually exclusive set of words for one or other label. This may not work in sentiment analysis where same word can have different meanings depending on surrounding words. N-Grams and then TF on it might help in that case.
I am familiar with logistic regression (studying to be an actuary) but the problem is that my documents are unlabeled: I literally have 2000+ unlabeled documents and a list of positive and negative words.
I'm willing to label 10% (~200) documents but how should I 'score' them? On a scale of [-1,1]? Just {-1,0,1} for negative, neutral, positive? How do I create a training set?
Also can you point me in the direction of some of the implementation details e.g. how do you translate text into logistic regression model?
I would also like to implement POS tagging and 2-grams (e.g. "not bad" != "bad") - any advice on incorporating this into the system?
Thank you for your input!