Re:Google Street View image of a house predicts car accident...
by londons_explore on Friday December 13, 2019 @08:15AM
(#21781220)
It doesn't impact the validity of the results of the study as far as I can see... If they got closer to a perfect rating system, the predictive power of the model could only be better.
If there are two raters in a sample of six that disagree with the four others, I would argue that the IRR is an issue. Otherwise one could always run studies with n experts and only select a subsample of those raters that have a high interrater agreement, or not?
The Fleiss kappas are already relatively low..