News for hackers. Stuff that matters.

A comment on New exponent functions that make SiLU and SoftMax 2x faster, at full accuracy.

Re:New exponent functions that make SiLU and SoftMax 2x fast... by lxe on Wednesday May 15, 2024 @06:34PM (#40373222)
At this point, is gguf/llama.cpp a more performant solution for unbatched inference on CUDA devices, or is exllamav2+flashattention still reigning supreme?
[ Parent | on HN ]
Re:New exponent functions that make SiLU and SoftMax 2x fast... by GuuD on Wednesday May 15, 2024 @06:59PM (#40373389)
The difference is negligible on 2x 4090. There are more important differences like 4 bit KV cache.
[ Parent | on HN ]