jetaim
One of the oldest and largest biomedical institutes, the Robert Koch Institute in Germany, recently had a few press releases, urging for tests for at least ALL respiratory tract infections.
What one of the leading experts, Prof. Drosten, also mentioned is that current (PCR) tests have considerable false positives. The effect of such FP at large scale can hardly be estimated.
I really hope that you are able to find a solution and can bring up a scalable and reliable solution, after the promises. If not, there will only remain the impression that this could be a Silicon Valley type of talk the walk, as people heard it from other companies in the past.
If there are two raters in a sample of six that disagree with the four others, I would argue that the IRR is an issue. Otherwise one could always run studies with n experts and only select a subsample of those raters that have a high interrater agreement, or not?
The Fleiss kappas are already relatively low..