News for hackers. Stuff that matters.
Re:The fraudulent claims made by IBM about Watson and AI by PaulHoule on Monday February 06, 2023 @08:23AM (#34676923)
That's exactly what I'm pointing out. If you look at Kaggle answers or arXiv papers you might not even think calibration was a thing, but it's a key tool when going from academic ML to industrial ML.

I got into calibration through text retrieval where the TREC methodology rates relevance functions by rank in such a way that you don't get points for being calibrated or well-calibrated.

There are numerous things mainstream text retrieval systems don't do, most notably it is hard to build an alerting feature without calibration. Practically you have to set some relevance threshold to avoid getting too much irrelevant stuff.

I calibrated a rather good search engine by fitting a curve to the score and found that the best it would ever give is p=0.7 to be relevant and that it gave that very rarely. So even with a calibrated score you wouldn't be able to set a very high threshold or expect to get many documents. Watson gets at this problem by having a large number of question answering models each which has a high p to be right when it is right but each of which also has very low recall, something you can do when using p as a universal score.

There are no comments on this comment yet.