News for hackers. Stuff that matters.

A comment on Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy. Parent comment

Re:Let's build GPT: from scratch, in code, spelled out by An... by rockwotj on Tuesday January 17, 2023 @10:38PM (#34423179)
You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links

The pretrained model is already available.

[ Parent | on HN ]
Re:Let's build GPT: from scratch, in code, spelled out by An... by tysam_and on Wednesday January 18, 2023 @03:08PM (#34432297)
GPT-J I think hasn't gone beyond 20B parameters, and while it is not the most obvious I think the original question is asking about the full 180B parameter+ kind of model. :) :thumbsup:
[ Parent | on HN ]