Fine-tuning is a much better proposition than you’re giving it credit for. Papers are coming out demonstrating that 7B parameter models can outperform GPT-4’s quality when trained on a limited set of tasks. Yet, a 7B model offers comparatively cheap and fast inference. Furthermore, for a lot of use cases, few-shot priority is infeasible because you need to supply 2-3k tokens worth of few-shot examples with every prompt in order to fully specify the behavior you want. (As an example, think of long-form summarization where you want the summary to adhere to certain rules.)