When it’s more feasible to do inference on the client (browser or desktop) I can see SLMs popping up more common in production.
If you want to scale up and down on demand, you can just fine tune on openai and google cloud as well.
I don't think that's true.
I can fine tune a model by renting a few A100s for a few hours, total cost in the double digit dollars. It's a one-time cost.
Running inference with the resulting model for a production application could cost single digit dollars per hour, which adds up to hundreds or even thousands of dollars a month on an ongoing basis.
That may or may not be true for use-cases that require asynchronous, bulk inference _and_ require some task-specific post-training.
FWIW, my approach towards tasks like the above is to
1. start with using an off-the-shelf LM API until
2. one figures out (using evals that capture product intent) what the failure modes are (there always are some) and then
3. post-train against those (using the evals)
It's not actually that expensive and hard. For narrow usecases, you can produce 4-bit quantized fine-tunes that perform as well as the full model. Hosting the 4-bit quantized version can be done on relatively low cost. You can use A40 or RTX 3090 on Runpod for ~$300/month.
In practice prompt engineering and few-shot prompting with modern LLMs, due to their strong-and-only-getting-better-over-time prompt adherence, tends to be more pragmatic.