News for hackers. Stuff that matters.

A comment on Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook. Parent comment

Re:Show HN: Alpaca.cpp – Run an Instruction-Tuned Chat-Style... by umangsh on Thursday March 16, 2023 @03:56PM (#35187506)
30B fp16 takes ~500 ms/token on M2 Max 96GB. Interestingly, that's the same performance as 65B q4 quantized.

65B fp16 is ungodly slow, ~300,000 ms/token on the same machine.

[ Parent | on HN ]

There are no comments on this comment yet.