Re:GPT-4 Details Leaked
by pas on Tuesday July 11, 2023 @09:09AM
(#36680452)
Interestingly Google was using ~2000 experts back in the first Trasnformer architecture (if I understand correctly) https://www.youtube.com/watch?v=9P_VAMyb-7k&t=6m42s [sparsely-gated mixture of experts layer]
[ Parent
| on HN ]
There are no comments on this comment yet.