Re:Ask HN: How to avoid AI Dungeon-style bills? I’d like to ...
by p1esk on Saturday January 04, 2020 @06:05PM
(#21958269)
Re quantization - you should be fine with INT8. So that's 1.5GB. And after pruning you will probably be able to compress it down to 500MB. To clarify - we are talking about distributing a trained model, right? You don't need to train it in reduced precision. Both TF and Pytorch provide tools to quantize a model after training.
[ Parent
| on HN ]
Re:Ask HN: How to avoid AI Dungeon-style bills? I’d like to ...
by sillysaurusx on Saturday January 04, 2020 @06:42PM
(#21958475)
Doesn't quantization harm model quality?
It's hard to quantify how much damage quantization does, but natural language generation is very subtle.
Re:Ask HN: How to avoid AI Dungeon-style bills? I’d like to ...
by p1esk on Saturday January 04, 2020 @06:53PM
(#21958539)
When you go below 16 bits during training the gradient descent might become unstable, and there might not be enough precision for small weight updates. But for inference, 8 bits should be enough. It mostly depends on the model size and if there's enough learning capacity for the dataset. I suspect GPT2 1.5B is very overparametrized, so in the future it will probably be considered similar to VGG, which is highly compressible without any accuracy loss. I wouldn't be surprised if you can finetune GPT2 using only 3-4 bits for both weights and activations, with no quality loss. By 'finetune' here I mean finetuning for quantization, not transfer learning.