News for hackers. Stuff that matters.
Re:OpenAI releases larger GPT-2 model by leod on Sunday May 05, 2019 @11:12AM (#19833187)
Thank you so much for your comprehensive answer, this helps a lot.

If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do you know if this works better than the learning rate schedule that is usually used for Transformer models (https://www.tensorflow.org/alpha/tutorials/text/transformer_...)?

Re:OpenAI releases larger GPT-2 model by gwern on Sunday May 05, 2019 @11:43AM (#19833342)
It's a constant, yes. We haven't tried any other learning rate schedules (for my poetry GPT-2s, I simply drop the LR 10x each day or so). I have no idea if this is optimal for transfer learning or not.