GPT-J is cool and all, but it’s been out for awhile and hasn’t seen anywhere near this amount of network effects. Further the Eleuther team is a bit detached from the “average user” in my experience, being all-in on TPU and JAX (which is still suffering from a similar lack of maturity making it difficult to run on non-standard setups).

I’m sure someone did the work you suggest to let folks run on this on more reasonable setups- but I never saw it on the front page here.

Did they achieve GPT-3 performance on anything at that model size? Happy to be corrected, but my understanding was llama does in fact outperform GPT-3 even at the smaller parameter-counts on certain downstream tasks due to using the chinchilla training regime (and other tricks).