You have to eyeball it because (a) 'what it looks like to a human eyeballing it' is the most important metric for misuse - no one cares that much about a model which has a slightly better log loss but a human can instantly spot is robospam, that can't be abused or used for fun projects like generating poetry; and (b) the log loss between GPT-2 and his will not be comparable due to the different training corpuses (and possibly architectural differences as well), so the actual metrics are not useful.

Given the large difference in quality, it only takes a few comparisons. Again, just look at the samples. Read a few. Do they really look the same in coherency and quality and realism?

Do you think that if you were given a sample of text genereated by one of the two systems you would be able to tell which system generated it?

Looking at two texts side-by-side while knowing which is which may well create an illusion of striking differences, which however are not really there. This is why concrete metrics are necessary- because human judgement is biased and inconsistent.

But, like you say- there are no good metrics. In fact, the only thing we have to go by when discussing Open AI's model in the first place is opinions and convictions. Not least the opinion of OpenAi that their model generates text of unprecedented coherence, which itself is not based on anything else than eyballing.

I think I could. Few or none of the new samples even manage to maintain coherency in a sample, and you can see all sorts of garbage prose being generated.

Perhaps you could. Personally, I think it's impossible to get an accurate comparison in this way.

I think it is possible, because it turns out I am right: the quality is much lower when compared directly on perplexity as well: https://medium.com/@NPCollapse/addendum-evaluation-of-my-mod...