Re:Choose Your Weapon: Survival Strategies for Depressed AI ...
by anothernewdude on Friday April 14, 2023 @06:45AM
(#35567889)
And? ChatGPT doesn't give you logprobs. It doesn't allow you to alter the generation algorithm. They've left their lunch out on the table for someone else to eat.
The trick is always to offset the cost of inference with larger cost of training. You don't apply the Chinchilla scaling law, that's for academics and people who don't have to pay for inference. You pretrain the model 10x longer (like 1T tokens for LLaMA) to make it the best you can fit into an A100 or 4090 quantised to 4 or 3 bits. So everyone can have AI assistants running on their own toys.