Re:GPT-2: 6-Month Follow-Up
by gwern on Tuesday August 20, 2019 @01:18PM
(#20749423)
No. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)
[ Parent
| on HN ]
Re:GPT-2: 6-Month Follow-Up
by JonathanFly on Tuesday August 20, 2019 @02:18PM
(#20750156)
On the NVIDIA GPT- 2 implementation:
>What would be the largest model one could train across 2x 2080Ti?
>~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations
They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).
Re:GPT-2: 6-Month Follow-Up
by JonathanFly on Tuesday August 20, 2019 @04:03PM
(#20751433)
I haven't look into at all myself, but he also said:
>We do provide training code that should work out of the box for gpt2 117M/345M
It would take forever (or $$$) to train even 117M model from scratch.
Re:GPT-2: 6-Month Follow-Up
by JonathanFly on Tuesday August 20, 2019 @05:02PM
(#20752107)
I read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.
Re:GPT-2: 6-Month Follow-Up
by The_rationalist on Tuesday August 20, 2019 @02:14PM
(#20750108)
You maybe should try tensorflow automatic mixed precision!
https://github.com/zihangdai/xlnet/pull/200
fp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.
Have you fine tuned 774 successfully using a single GPU?