I'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).

Ouch! So 11GB is nowhere close to being enough, then. I wonder if even switching to FP16 will be adequate?

Might be able to get 745M down to work on a single GPU. I'm definitely not using all 24GB, so fp16 might be able to get it down enough.

How would you use fp16 to get it to work on a single GPU? And if you did, what GPU should you use?

Figured. I'll make changes to allow sampling from the default model more easily.

Are you using FP16?

No. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)

On the NVIDIA GPT- 2 implementation:

>What would be the largest model one could train across 2x 2080Ti?

>~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations

https://twitter.com/TheRealRPuri/status/1161322580126470145

They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).

I haven't look into at all myself, but he also said:

>We do provide training code that should work out of the box for gpt2 117M/345M

https://twitter.com/TheRealRPuri/status/1161319745259393024

It would take forever (or $$$) to train even 117M model from scratch.

I read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.

You maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200

fp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.

Have you fine tuned 774 successfully using a single GPU?

I recommend Nvidia Apex, it offers several ways to mix precision.