Re:80% faster, 50% less memory, 0% loss of accuracy Llama fi...
by danielhanchen on Friday December 01, 2023 @09:18PM
(#38495224)
Ohhh interesting! So like ML Perf or Faster CPython https://github.com/faster-cpython/benchmarking-public?
Re:80% faster, 50% less memory, 0% loss of accuracy Llama fi...
by kristopolous on Friday December 01, 2023 @09:36PM
(#38495338)
Sure. This thing I propose might already exist in a nascent form.
I haven't checked in a few months and the way things are moving now...
Re:80% faster, 50% less memory, 0% loss of accuracy Llama fi...
by danielhanchen on Friday December 01, 2023 @09:49PM
(#38495412)
Interesting!! I'll talk with my brother about this - sounds very cool!
Re:80% faster, 50% less memory, 0% loss of accuracy Llama fi...
by ptd on Saturday December 02, 2023 @03:23AM
(#38496985)
Hey Daniel, I would love to help out on this. I'm learning about LLMs and this benchmarking project sounds like a fun way to further my knowledge and skills. I sent you a message on LinkedIn.
Cheers!
Re:80% faster, 50% less memory, 0% loss of accuracy Llama fi...
by danielhanchen on Saturday December 02, 2023 @03:40AM
(#38497078)
Cool lets chat on Linkedin! :)
For instance, are any of your prompting tests in say, Korean? What about winograd schema challenges in languages other than English? Japanese for instance, comes with its own unique set of context ambiguities that do not appear in English. I'm sure dozens of languages are similar. It'd be nice to have user contributable tests to cover the breadth of use cases here.
A great optimization that moves a score let's say from 95% -> 5% on "winograd-persian" may be fine or may be a show stopper, depends on what you care about.
That's why it's gotta be normalized, future-proof, and crowdsourced.