Re:A Code Glitch May Have Caused Errors in More Than a Hundr...
by bonoboTP on Saturday October 12, 2019 @07:25PM
(#21237060)
It often feels like a quixotic fight. Strong claims sell papers, but are hard to stand behind for a researcher with high standards, who is very critical of overall practices in a field. Their "best" recourse in the idealistic sense would be to publish an overarching meta analysis of the field, and pointing out problematic practices, gaining enemies and being labeled as a crusader.
In my field, computer vision, a major problem is uncomparable setups, far reaching claims etc. A major pattern is introducing some fancy model, showing that it gets a bit better results that could well be due to noise or some tuning on the test set (not even secretly, just trying many different things and putting the bold formatting on your best number and claiming it as SOTA). Whether it was really due to your fancy model or not is hard to prove. But my experience is that mundane, pedestrian changes in a model introduce way bigger performance changes than whether or not you use the fancy module or architectural tweak that is proposed in mediocre published works. Which also means you can absolutely not compare models created by different people, especially when the performance gap is small. Tiny details can screw up the whole ordering, so trying to interpret the order and draw conclusions from it is equivalent to reading tea leaves.
Now surely this applies mostly to mediocre research. But that is the majority. Top of the line research is more solid, but if you're a small unknown researcher, your best strategy is to start getting out there, publish and network.
In my field, computer vision, a major problem is uncomparable setups, far reaching claims etc. A major pattern is introducing some fancy model, showing that it gets a bit better results that could well be due to noise or some tuning on the test set (not even secretly, just trying many different things and putting the bold formatting on your best number and claiming it as SOTA). Whether it was really due to your fancy model or not is hard to prove. But my experience is that mundane, pedestrian changes in a model introduce way bigger performance changes than whether or not you use the fancy module or architectural tweak that is proposed in mediocre published works. Which also means you can absolutely not compare models created by different people, especially when the performance gap is small. Tiny details can screw up the whole ordering, so trying to interpret the order and draw conclusions from it is equivalent to reading tea leaves.
Now surely this applies mostly to mediocre research. But that is the majority. Top of the line research is more solid, but if you're a small unknown researcher, your best strategy is to start getting out there, publish and network.