Re:Help EFF Track the Progress of AI and Machine Learning
by daveguy on Wednesday June 21, 2017 @04:30PM
(#14607628)
I think you'd have to measure "common sense reasoning" first. With NLP you can have accuracy/precision scores. With common sense it's almost equivalent to "what you need to beat the turning test" and not easily measured. I think if you get that part you essentially have the NLP. In that those last few percent error of NLP are generally "common sense" issues.
So I think you need some kind of "meta-metric" that measures the growth of the taxonomy itself. And perhaps some kind of weighting for the impact of the solution.
There is also an interaction effect (for instance, Natural Language Processing is powerful, and "common sense reasoning" is powerful, but put them together and you have a knockout), but I don't know how to go about measuring that.