Batching: Your throughput (and thus cost per token) will be much worse if you don't do batching (meaning running inference for several inputs at once). To do batching, you either need to have a high sustained QPS or adjust your workload to be bursty (but this has utilization issues).
Inference engine: The regular HuggingFace, even with built in optimizations, is not competitive against inference engines like vllm, exllama, etc.
Utilization: Depending on your scale, it might be hard for you to have a nice flat utilization, or to even utilize one GPU at 100% capacity. This means you need to solve scaling up & down your machines and the issues associated with it (can you live with cold starts? etc.)
Hardware: Forget server-less GPUs like Replicate etc, their markups compared to on-demand pricing is usually >10x. On-demand A100 or H100, provided you can even get a quota for them, are also expensive. Spot instances are better. Older GPUs (A10, T4, etc.) are better. Your own GPU cluster is likely the best if you have the scale (and likely, you can resell your cluster within the next 18months without much if any depreciation).
For these reasons, I have been toying with the idea of providing a dead simple service for fine-tuning & serving open-source LLMs where the users actually own (and can download) the weights. If anyone is interested in this and would like to chat, let me know.
It could be a layer on top of $gpu_cloud (so end user signs up for the cloud) and the service makes use of those resources.
Our special case is a single model, high throughput and latency unsensitive.
Can we use multiple lower-memory GPUs, split the the model "horizontally", and pipeline each batch/inference across the GPUs?
Is this a common scenarios handled by the engines you mentioned ?
How do you do that? Could you please provide a link if you have it handy.
Of course, speed, accuracy, power consumption, etc are all considerations, sometimes at odds with each other.
What EXL2 seems to bring to the table is that you can target an arbitrary quantize bit-weight (eg, if you're a bit short on VRAM, you don't need to go from 4->3 or 3->2, but can specify say 3.75bwp). You have some control w/ other schemes by setting group size, or with k-quants, but EXL2 is definitely allows you to be finer grained. I haven't gotten a chance to sit down with EXL2 yet, but if no one else does it, it's on my todo-list to be able to do 1:1 perplexity and standard benchmark evals on all the various new quantization methods, just as a matter of curiosity.
Saying it can outperform GPT-3.5-turbo on "many tasks" would be venturing into unreasonable territory.
Saying it can outperform GPT-3.5-turbo or even GPT-4 on many tasks is just setting people up for disappointment: There's a reason ARC (common sense reasoning) never gets mentioned when touting OS LLMs vs commercial, and the gap in ARC performance is amplified terribly when you try to apply chain-of-thought
What do you mean by evaluation?
https://paperswithcode.com/sota/common-sense-reasoning-on-ar...
GPT-3 53.2
GPT-3.5 85.2
LLaMa-65B 56.0
And surprisingly, Llama 33B performs _better_ than Llama 65B!
TigerResearch/tigerbot-70b-chat: ARC (76.79)
Still below 3.5 and like most top entries no clear training objectives, which usually means they were fine-tuned on the benchmark itself.
-
ARC is easily the most important benchmark right now for widespread adoption of LLMs by laypeople: it's literal grade school multiple choice, created to the bar of an 8th grader.
When you sit people down in front of an LLM and have them interact with it: ARC has by far the closest correlation of how "generally smart" the model will feel.
This may be acceptable for some use cases, bot not for others where even regular gpt-3.5 may not be economical.
On many tasks, fine-tuned Llama can outperform GPT-3.5-turbo or even GPT-4, but with a naive approach to serving (like HuggingFace + FastAPI), you will have hard time beating the cost of GPT-3.5-turbo even with the smaller Llama models, especially if your utilization is low.