News for hackers. Stuff that matters.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by lidHanteyk on Thursday October 24, 2019 @10:07PM (#21351172)
Yes. This is more generally known as Goodhart's Law[0]: when a metric is used as a goal, then people will game the metric in order to win, making the metric useless.

There is no fundamental way to overcome this problem, except by not using metrics as goals.

[0] https://en.wikipedia.org/wiki/Goodhart's_law

There are no comments on this comment yet.