The embedding system uses a probability-calibrated SVM. My average AUC is 0.77, I hear TikTok gets in the low 80’s and they are using collaborative filtering. I got 0.72 with a bag-of-words and logistic regression model.
From a product standpoint it’s got the disadvantage that it takes about 1000 judgements to really get good, right now I am training over the last 40 days of data because it doesn’t really get better with more than that which is good news because the compute and storage are nicely bounded.
On an unrelated note I realized recently that the 'bag' in bag-of-words is another name for the multiset data structure... Which makes sense when you think about the text as being a _set_ of tokens which can appear _multiple_ times.
I was talking to somebody about the potential as an open source project and came to the conclusion that it's a research project right now but my research projects are more solid than average. I know I'm not afraid to demo it because I run it every day and it spins like a top.
If you want to chat about it look up my profile and send me an email.
Interested in your embedding based system - is that embedding layer + neural net?
Sounds very cool overall:)