News for hackers. Stuff that matters.
Re:Google Street View image of a house predicts car accident... by jetaim on Friday December 13, 2019 @03:58AM (#21780138)
One point that confuses me a bit: The study initially used experts (no word on the experts?) and later only selects four experts for the subsample because their kappa is moderate. Is that a common approach?

If there are two raters in a sample of six that disagree with the four others, I would argue that the IRR is an issue. Otherwise one could always run studies with n experts and only select a subsample of those raters that have a high interrater agreement, or not?

The Fleiss kappas are already relatively low..

Re:Google Street View image of a house predicts car accident... by londons_explore on Friday December 13, 2019 @08:15AM (#21781220)
It doesn't impact the validity of the results of the study as far as I can see... If they got closer to a perfect rating system, the predictive power of the model could only be better.