A Python Library to 6-7x the inference speed of your HF models

Re:A Python Library to 6-7x the inference speed of your HF m... by behnamoh on Thursday February 22, 2024 @12:28AM (#39463578)

does anyone use transformers in production? last I checked people either use exllama2, vLLM, or llama.cpp. AFAIK llama.cpp already does speculative decoding (even batched grammar based spec. decoding!).

[ Parent | on HN ]

Re:A Python Library to 6-7x the inference speed of your HF m... by daemonologist on Thursday February 22, 2024 @01:19AM (#39463798)

We use transformers in production for T5 models, which aren't supported by the usual suspects. I did a bit of poking around and might try out TensorRT or CTranslate2 at some point; for now the single GPU we have dedicated to the project has been able to keep up with incoming data, but I'll probably have to address it in the next couple of months.

If anyone has suggestions for other T5 inference options I would welcome them.

[ Parent | on HN ]

Re:A Python Library to 6-7x the inference speed of your HF m... by iAkashPaul on Thursday February 22, 2024 @04:03AM (#39464796)

Indirectly via TGI

[ Parent | on HN ]

Re:A Python Library to 6-7x the inference speed of your HF m... by jph00 on Thursday February 22, 2024 @03:36AM (#39464660)

vLLM uses transformers.

[ Parent | on HN ]

Re:A Python Library to 6-7x the inference speed of your HF m... by dmezzetti on Thursday February 22, 2024 @05:13AM (#39465221)

It's important to note that Transformers can be used for a lot more than just LLMs. There are plenty of models that can be fine-tuned to perform a single task better than LLM prompting.

[ Parent | on HN ]