Re:How Is LLaMa.cpp Possible?
by pocketarc on Wednesday August 16, 2023 @05:37AM
(#37144694)
I'd be more inclined to believe that they're dropping down to gpt-3.5-turbo based on some heuristic, and that's why sometimes it gives you "dumber" responses. If you can serve 5/10 requests with 3.5 by swapping only the "easy" messages out, you've just cut your costs by nearly half (3.5 is like 5% of the cost of 4).
[ Parent
| on HN ]