News for hackers. Stuff that matters.
Re:Help EFF Track the Progress of AI and Machine Learning by startupdiscuss on Wednesday June 21, 2017 @03:10PM (#14606974)
Part of what we are learning, in AI/ML, is the number of problems where the ML is relevant and where it is not. (Or, to put it another way, where we can find statistical relevance between features and targets).

So I think you need some kind of "meta-metric" that measures the growth of the taxonomy itself. And perhaps some kind of weighting for the impact of the solution.

There is also an interaction effect (for instance, Natural Language Processing is powerful, and "common sense reasoning" is powerful, but put them together and you have a knockout), but I don't know how to go about measuring that.

Re:Help EFF Track the Progress of AI and Machine Learning by daveguy on Wednesday June 21, 2017 @04:30PM (#14607628)
I think you'd have to measure "common sense reasoning" first. With NLP you can have accuracy/precision scores. With common sense it's almost equivalent to "what you need to beat the turning test" and not easily measured. I think if you get that part you essentially have the NLP. In that those last few percent error of NLP are generally "common sense" issues.
Re:Help EFF Track the Progress of AI and Machine Learning by pde3 on Thursday June 22, 2017 @03:30AM (#14610693)
Yes, we've also thought about the "meta metric" idea. We may try to make one if we feel that the taxonomy is at all complete and trustworthy.