Paving the way to efficient architectures: StripedHyena-7B

Re:Paving the way to efficient architectures: StripedHyena-7B by firejake308 on Friday December 08, 2023 @05:25PM (#38575480)

Darn, I was hoping the RWKV people had finally obtained reportable results. This is still interesting, though. Maybe we will see more alternatives to transformers soon

[ Parent | on HN ]

Re:Paving the way to efficient architectures: StripedHyena-7B by senseiV on Friday December 08, 2023 @08:18PM (#38577269)

They Do, the latest rwkv v5, matches mamba at 3b scale, and from the benchmarks I see, its similar to hyena

[ Parent | on HN ]

Re:Paving the way to efficient architectures: StripedHyena-7B by stellaathena on Friday December 08, 2023 @07:32PM (#38576890)

RWKV had a paper accepted at EMNLP and released models which match the performance of equivalent transformers.

What else are you looking for?

[ Parent | on HN ]

Re:Paving the way to efficient architectures: StripedHyena-7B by refulgentis on Friday December 08, 2023 @06:39PM (#38576393)

There are RWKV numbers in there, so you sort of got your wish sideways. :)

Has RWKV not released anything until now? I thought it was an open project that was in use, if sort of by a hipster 1%

[ Parent | on HN ]

Re:Paving the way to efficient architectures: StripedHyena-7B by stellaathena on Friday December 08, 2023 @07:34PM (#38576910)

Paper: https://arxiv.org/abs/2305.13048

Models (v4 are the ones from the paper): https://huggingface.co/RWKV

GitHub: https://github.com/BlinkDL/RWKV-LM

[ Parent | on HN ]

Re:Paving the way to efficient architectures: StripedHyena-7B by senseiV on Friday December 08, 2023 @08:19PM (#38577287)

V5 7b is out, close to hyena, gets 1400 t/s on a 3090, while an h100 llama 7b 8bit is 1200 t/s

[ Parent | on HN ]