News for hackers. Stuff that matters.

A comment on A guide to open-source LLM inference and performance.

Re:A guide to open-source LLM inference and performance by bicepjai on Monday November 20, 2023 @06:38PM (#38356594)
That’s really detailed explanation. Can we do something like this for M1 ultra/M2 ultra/M3 max with large RAM ?
[ Parent | on HN ]
Re:A guide to open-source LLM inference and performance by varunshenoy on Monday November 20, 2023 @06:54PM (#38356755)
Absolutely. Looks like the M1 Ultra has 800GB/s of memory bandwidth and ~20 TFLOPS of compute.

The same calculations from the post should hold, except with these new values.

[ Parent | on HN ]