News for hackers. Stuff that matters.
Re:Ask HN: How to avoid AI Dungeon-style bills? I’d like to ... by p1esk on Saturday January 04, 2020 @06:53PM (#21958539)
When you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for inference, 8 bits should be enough. It mostly depends on the model size and if there's enough learning capacity for the dataset. I suspect GPT2 1.5B is very overparametrized, so in the future it will probably be considered similar to VGG, which is highly compressible without any accuracy loss. I wouldn't be surprised if you can finetune GPT2 using only 3-4 bits for both weights and activations, with no quality loss. By 'finetune' here I mean finetuning for quantization, not transfer learning.

There are no comments on this comment yet.