Re:Show HN: Speeding up LLM inference 2x times (possibly)
by kolinko on Wednesday April 17, 2024 @06:09PM
(#40070590)
The algorithm still uses all the weights, just not all the time - just skips the weights when they are not important given an input vector.
Also, approximation methods, as a field, are not new and they have shown their use.
Having said all that, extraordinary claims require extraordinary evidence - that’s why I hedge the communication messages. It’s „probably” until we get serious tests going on
Also, approximation methods, as a field, are not new and they have shown their use.
Having said all that, extraordinary claims require extraordinary evidence - that’s why I hedge the communication messages. It’s „probably” until we get serious tests going on