Show HN: Route your prompts to the best LLM

Re:Show HN: Route your prompts to the best LLM by treprinum on Thursday May 23, 2024 @05:42AM (#40452590)

Does the router approach make sense with multi-tasking LLMs? One could execute multiple different tasks with a single prompt (chat response, NER, translation etc.) and with the latest models even do images or video alongside text. Doesn't a router get in the way, unnecessarily increasing latency?

[ Parent | on HN ]

Re:Show HN: Route your prompts to the best LLM by danlenton on Thursday May 23, 2024 @06:27AM (#40452908)

So the neural scoring introduces ~20ms latency, but this only impacts time-to-first-token (not inter-token-latency). When using our public endpoints there is an additional ~150ms latency, but you can deploy the router on-prem in your own cloud, so then it would only be the inference latency. Generally the improvements in ITL outweigh the small addition to TTFT.

[ Parent | on HN ]