News for hackers. Stuff that matters.

A comment on GPT-J-6B: 6B JAX-Based Transformer. Parent comment

Re:GPT-J-6B: 6B JAX-Based Transformer by homedepotdave on Thursday June 10, 2021 @09:57AM (#27460814)
Out of curiosity, how fast are your inferences with this setup?
[ Parent | on HN ]
Re:GPT-J-6B: 6B JAX-Based Transformer by Ari_Rahikkala on Thursday June 10, 2021 @11:50AM (#27462098)
With the defaults of per_replica_batch=1, seq=2048 and gen_len=512, a completion takes about 20 seconds.

I'm not sure yet what settings I'll end up with if I decide to play with this more. per_replica_batch=3, seq=1024, gen_len=64 would give an experience roughly similar to the AI Dungeon that I'm used to, though less clever than the Dragon model, and a bit slower at about 10 seconds per batch.

[ Parent | on HN ]