So we can't lay eggs yet, at least we got one. Who knows what might hatch.
I guess I just felt an objection about “[oss] running circles around OpenAI.” The reality is that the core technology is still very unapproachable (making these models). OSS solved like 1% of the total problem/difficulty.
My biggest hope is that meta/google/anthropic actually decide to release training code + data + checkpoint. Meta is most likely to do it but still unlikely
Without that it would take eutherai/carperai to really exceed expectations for OSS to be competitive.
There are only a few hundred folks who have actual experience training at this scale.
llama is both not oss and not as good as da-vinci-003/etc. Prob best actual oss model is like flan-t5-xxl.