Re:How to run Llama 13B with a 6GB graphics card
by snovv_crash on Sunday May 14, 2023 @11:25AM
(#35938675)
An interesting corollary of this is that if you want to reduce the model size you can compensate by training for longer to achieve the same accuracy. Depending on your training:inference ratio this may be more optimal globally to reduce your total compute costs or even just reduce your frontend latency.
Further reading: https://dynomight.net/scaling/