At my job, I can’t casually fire up 8xA100 80gb instances. And if I could, the performance wouldn’t have the throughput I require to be useful. Big models are operationally much more expensive.
The smallest/fastest model that is accurate enough for your use case is ideal.
Sure.
…but it’s also fair to say that the smallest model that can fit your use case will be bounded by the parameter count.
No amount of training data can make 100 param model do text summarisation.
If you have a 3B param model, and you want a chat-GPT to embed in your app, do you think it’ll do?
I don’t.
The output is not at that quality level, because it’s too small.
Not everyone needs that; but these 3B / 7B models don’t have the capability to do everything.
Sounds like the difference between edge and centralized ML scoring.
But ultimately small models are very good for most things, and much more preferable (to run at the home to organize your digital life, with a small SBC or old computer)
So we are all in agreement here that a 3B model is fundamentally inferior to a larger model?
Not that it doesn’t have uses; not that there’s no value in research in small models.
Just, honestly, that these smaller models don’t have the capabilities of the larger models.
It’d be good to be a direct acknowledgment of that, because it seems like you’re going out of your way to promote the “it’s fine to have a small model”; and it is, Rosslyn speaking. Parameter count isn’t everything. Small models are accessible, you can easily fine tune them. They are interesting.
…but, they are not as good, as far as I’m aware, in terms of output, as larger models.