Re:Bad numbers in the “gzip beats BERT” paper?
by p1esk on Monday July 17, 2023 @11:54AM
(#36759744)
Did you try contacting the authors before you went public with your discovery?
[ Parent
| on HN ]
We're adult enough to have discussions like this in public. They are healthy to have. People make mistakes. Kudos to the original authors for releasing the source code so people could inspect and replicate their results.
Re:Bad numbers in the “gzip beats BERT” paper?
by returningfory2 on Monday July 17, 2023 @02:18PM
(#36761942)
I agree, and just want to add: nowadays it's super common for researchers to widely publicize their new work on social media. The blog post here even mentions "Table 5 from the paper was often included in tweets".
In this context of sharing your results very publicly, it seems only fair that rebuttals would be very public, too. Otherwise researchers would be highly incentivized to very publicly publish weak results because they would get a lot of positive publicity when they share the results, but not much negative publicity when the weaknesses are found and shared.
Re:Bad numbers in the “gzip beats BERT” paper?
by jarym on Monday July 17, 2023 @12:44PM
(#36760507)
It isn't a security issue and doesn't warrant responsible disclosure so why would op be expected to?
Re:Bad numbers in the “gzip beats BERT” paper?
by ks2048 on Monday July 17, 2023 @03:48PM
(#36763101)
I did not, but I see why that could be a better approach. I mainly am trying to be more open with little side projects I do, so wanting to start blogging what I'm working on. Also, this paper was beiung widely discussed so thought this would be one more entry in that.