News for hackers. Stuff that matters.

A comment on GPT-3 is no longer the only game in town. Parent comment

Re:GPT-3 is no longer the only game in town by totoglazer on Monday November 08, 2021 @06:56PM (#29155958)
What library did you use for JAX on TPU? Also curious how much data you had?
[ Parent | on HN ]
Re:GPT-3 is no longer the only game in town by rg111 on Tuesday November 09, 2021 @12:08PM (#29164306)
We used Flax, of course.

We thought Haiku was cool, but none of us knew it, and learning resources were scarce. But we are very happy with Flax.

We used the Bengali subset of the mC4 dataset [0] for training the GPT-2 model.

[0]: https://huggingface.co/datasets/mc4

[ Parent | on HN ]