Re:GPT2, Counting Consciousness and the Curious Hacker
by gwern on Sunday June 09, 2019 @10:01AM
(#20139235)
I think I could. Few or none of the new samples even manage to maintain coherency in a sample, and you can see all sorts of garbage prose being generated.
Re:GPT2, Counting Consciousness and the Curious Hacker
by YeGoblynQueenne on Monday June 10, 2019 @04:20PM
(#20150420)
Perhaps you could. Personally, I think it's impossible to get an accurate comparison in this way.
Re:GPT2, Counting Consciousness and the Curious Hacker
by gwern on Thursday June 13, 2019 @10:13AM
(#20175027)
I think it is possible, because it turns out I am right: the quality is much lower when compared directly on perplexity as well: https://medium.com/@NPCollapse/addendum-evaluation-of-my-mod...
Looking at two texts side-by-side while knowing which is which may well create an illusion of striking differences, which however are not really there. This is why concrete metrics are necessary- because human judgement is biased and inconsistent.
But, like you say- there are no good metrics. In fact, the only thing we have to go by when discussing Open AI's model in the first place is opinions and convictions. Not least the opinion of OpenAi that their model generates text of unprecedented coherence, which itself is not based on anything else than eyballing.