News for hackers. Stuff that matters.
Re:RWKV RNN: Better than ChatGPT? by int_19h on Friday March 24, 2023 @06:54PM (#35296775)
It could go even further, in theory. The kind of ops that the current crop of LLMs needs is very simple, and at the same time there's no hard requirement for precision (which is why 4-bit quantization works so well). This means that unconventional approaches such as analog computing are potentially in the play again - it's easy to do addition and multiplication in an analog circuit, if you don't care about the answer being precise, and in theory one could pack a lot more of those circuits in the same space.
Re:RWKV RNN: Better than ChatGPT? by smaddox on Saturday March 25, 2023 @03:58PM (#35306625)
There's already at least one company with a product: Mythic AMP https://mythic.ai/products/m1076-analog-matrix-processor/

And there's several research papers about other architectures. For example: https://arxiv.org/abs/2205.05853