Re:GPT-4 is getting worse over time, not better
by kippinitreal on Wednesday July 19, 2023 @11:38AM
(#36788184)
Clever idea. I think you would have to recompute the context (ie embed the prior tokens) every time you swapped models because the weight distributions would be different for each model. Going from big->small might make this overhead worth it, but going back from small->big would assuredly be very costly.
Re:GPT-4 is getting worse over time, not better
by reportgunner on Wednesday July 19, 2023 @11:17AM
(#36787861)
So you run a model to avoid running a model? Math doesn't seem to add up.
Re:GPT-4 is getting worse over time, not better
by philovivero on Wednesday July 19, 2023 @11:46AM
(#36788319)
Running a model isn't binary, it's per amount of time spent generating tokens.
Re:GPT-4 is getting worse over time, not better
by GPUboy on Wednesday July 19, 2023 @12:18PM
(#36788864)
This is the topic of the language models as tool makers paper https://arxiv.org/pdf/2305.17126.pdf
Ie. easy tokens can be provided by a cheap-to-run model, and hard tokens are given by an expensive to run model?
A model could be used to decide when it is worth running the expensive model, based on the inputs, output so far, and probability distribution of the output of the cheap model.
For example, "Q: If I have 3 bananas and eat none, then how many bananas do I have?"
"A: You would have 3 bananas left, since you started with 3 and didn't eat any"
The "3" would come from the big model, while the rest all came from a small model.