News for hackers. Stuff that matters.
Re:ZML - High performance AI inference stack by Palmik on Tuesday September 17, 2024 @11:55PM (#41575624)
Given that the focus is performance, do you have any benchmarks to compare against the likes of TensoRT-LLM.
Re:ZML - High performance AI inference stack by gwenzek on Wednesday September 18, 2024 @03:39AM (#41576897)
It' s a bit early to compare directly to TensorRT because we don't have a full-blown equivalent.

Note that our focus is being platform agnostic, easy to deploy/integrate, good performance all-around, and ease of tweaking. We are using the same compiler than Jax, so our performances are on par. But generally we believe we can gain on overall "tok/s/$" by having shorter startup time, choosing the most efficient hardware available, and easily implementing new tricks like multi-token prediction.

Re:ZML - High performance AI inference stack by koe123 on Wednesday September 18, 2024 @03:48AM (#41576945)
I second this, it would help to justify the time investment into a framework if its clear how it stacks up!