Re:ZML - High performance AI inference stack
by Palmik on Tuesday September 17, 2024 @11:55PM
(#41575624)
Given that the focus is performance, do you have any benchmarks to compare against the likes of TensoRT-LLM.
[ Parent
| on HN ]
Re:ZML - High performance AI inference stack
by gwenzek on Wednesday September 18, 2024 @03:39AM
(#41576897)
It' s a bit early to compare directly to TensorRT because we don't have a full-blown equivalent.
Note that our focus is being platform agnostic, easy to deploy/integrate, good performance all-around, and ease of tweaking. We are using the same compiler than Jax, so our performances are on par. But generally we believe we can gain on overall "tok/s/$" by having shorter startup time, choosing the most efficient hardware available, and easily implementing new tricks like multi-token prediction.