News for hackers. Stuff that matters.

A comment on Show HN: Arch-Router – 1.5B model for LLM routing by preferences, not benchmarks.

Re:Show HN: Arch-Router – 1.5B model for LLM routing by pref... by pseudosavant on Tuesday July 01, 2025 @08:02PM (#44439077)
Not that LLMs are terribly latency sensitive (you wait on a lot of tokens), but what kind of latency impact does this have on requests that go through the proxy?
[ Parent | on HN ]
Re:Show HN: Arch-Router – 1.5B model for LLM routing by pref... by cotran2 on Tuesday July 01, 2025 @08:19PM (#44439167)
The model is compact 1.5B, most GPUs can serve it locally and has <100ms e2e latency. For L40s, its 50ms.
[ Parent | on HN ]
Re:Show HN: Arch-Router – 1.5B model for LLM routing by pref... by adilhafeez on Tuesday July 01, 2025 @08:25PM (#44439208)
Short answer is latency impact is very minimal.

We use envoy as request handler which forwards request to local service written in rust. Envoy is proven to be high performance, low latency and highly efficient on request handling. If I have to put a number it would be in single digit ms per request. I will have more detailed benchmark in the coming days.

[ Parent | on HN ]