News for hackers. Stuff that matters.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by Taek on Monday March 13, 2023 @10:14PM (#35146456)
Don't need a macbook anymore, any AVX2 CPU can do more than one token per second.

You could spend $600 and build your own PC that rubs LLaMA 65b.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by recuter on Monday March 13, 2023 @11:09PM (#35146906)
It is still much faster/cost-effective thanks to the unified memory. Would be interested in building out a PC as soon as something comparable becomes available.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by Taek on Monday March 13, 2023 @11:58PM (#35147276)
If you are going to spend $3500 on macbook because its more cost effective, might as well go all the way to buying an a6000 for $4095 which is iirc 5-10x faster than macbook
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by recuter on Tuesday March 14, 2023 @12:10AM (#35147343)
I am agnostic about Mac/PC but people sure like to chime in with these sort of comments.

First of all it is $4,700 on Newegg and secondly it is out of stock. This also leaves me with the rest of the computer to put together which is not $0. And finally it would absolutely not be 10x faster for this use case than a 64GB MBP.

The advantage of going with Apple is that I can pick one up immediately, use it as a laptop (soon with Asahi linux when it matriculates fully) and if need be easily sell it. You might not see the value in that, that's alright, people have different use cases and needs.