barneso
Presumably the task had been split up in the past to work around performance limitations of Hive; the fact that they got better performance in a single step shows that the tooling is improving and much less complexity is required to implement this kind of job.
I've always been tempted to do something with ML and the datasets I collect off my own body, but I have never seen this point. In my experience, the thing that allows for good diabetic control is a doctor who can help a person understand their own body's reaction as a dynamic system, rather than a rigid schedule or an external system that tells them what to do. A product like this would be most useful in my opinion as a learning aid.
The place I see this kind of data being extremely useful is in the insurance industry. I currently pay a 400% premium on my life insurance due to being diabetic, but my control is extremely good. Were it to allow me to obtain reasonable insurance coverage, I'd be more likely to be willing to go through the data collection headaches involved with this kind of product.
It requires a fair amount of set-up, but works surprisingly well once there is a core team and problems established.
We are building mldb.ai to help bring the data and the algorithms for ML together in a less ad-hoc manner and to help move things out of research and into prod once they are ready. Many of the hosted ML solutions (Azure ML, Amazon ML, Google Data Lab, etc) and other toolkits (eg Graphlab) are working on similar ML workflow and organizational structure problems.
Doing services is a reasonable response to that market dynamic, although they certainly had a product vision too.
Without knowing any of the details, I'd say they can be proud of what was accomplished and how they managed their exit: 1. From what I've heard, some of the work they did in the medical domain really did help save lives or make them better 2. They probably returned the capital to their investors and kept most of their employees 3. They didn't get bought by a company that would be horrible to work for or is in terminal decline
Credit where it's due: they entered a rough market and managed it well.
(edit: my grammar is good not)
There is a whole other world of non stochastic gradient descent based algorithms out there; IMO Tensorflow is sensible to stick to one class of algorithms and do it well.
(Disclaimer: I work on mldb, one of the tools on that list).
MLDB can memory-map some kinds of datasets which would also help with the low memory-to-datasest size ratio.
Please feel free to reach out (jeremy at datacratic) if you'd like to discuss further.
Could you describe your use-case? This is an interesting question and I'd love to hear more about what you are thinking of.
There are two main parts to most machine learning workloads: training and prediction (though in online scenarios they can be mixed). The training is what tends to be memory intensive, as it involves looking at historical data. Prediction, especially in real-time, is typically less memory intensive and more suited to your scenario. It is possible with MLDB to have training and prediction on separate instances to optimize hardware cost.
Our focus to date has been on getting large real-world use cases to fit, for both training and prediction, on a single commodity server with decent amounts of RAM (4GB and up), especially for training. Performance there is good; you can train user scorers or recommenders over hundreds of millions of users with 20 billion historical actions in minutes on a single server. We haven't focused much on the absolute memory footprint and there is some overhead; you will need a server with a good amount of memory, a couple of gigs, to have a pleasant user experience.
(edit): The team tells me it will work decently down to 256MB of RAM or so, for carefully designed automated workloads. For interactive use you will want more RAM.
I did misunderstand that you don't have a training set, just a list of positive and negative words. You could still apply a similar idea.
You could test your hypothesis that the score is biased by looking at the average number of positive and negative words per document, and slightly modify your factors. For example if you found that the average document had 6 negative words and 4 positive words, but you think that the average sentiment is neutral across your documents, you could multiply the positive word count by 1.5. It's a less brutal way to accomplish a similar outcome without increasing the complexity of your algorithm.
Otherwise, you will need to use an algorithm that has more discrimination power, and this will likely mean you need a training set. You can go very deep down that rabbit hole, but I would consider starting with Naive Bayes which is essentially learning a weight per positive and negative word and combines them in a similar manner to how you're doing so now. It has the advantage of being a simple algorithm.
1. Insert each negative example six times into your training set (or weight negative examples accordingly, ie use #positive matches - 6 * #negative matches / (2 * positive word count) as your score
2. Take your distribution of sentiment scores as calculated over held out data (or the training set itself, but be warned that this will skew your results), and calculate the mean and standard deviation. Normalize your results by subtracting the mean and dividing by the standard deviation. You can then say that positive sentiment is > 0 and negative sentiment < 0, with the absolute value being the strength of the classification.