Apple M3 Ultra - Hackdot

Re:Apple M3 Ultra by cubefox on Wednesday March 05, 2025 @11:10AM (#43268484)

> Transformers are typically memory-bandwidth bound during decoding.

Not in case of language models, which are typically bound by memory size rather than bandwidth.

[ Parent | on HN ]

Re:Apple M3 Ultra by whimsicalism on Wednesday March 05, 2025 @11:13AM (#43268535)

nope

[ Parent | on HN ]

Re:Apple M3 Ultra by cubefox on Wednesday March 05, 2025 @05:41PM (#43273743)

I assume even this one won't run on an RTX 5090 due to constrained memory size: https://news.ycombinator.com/item?id=43270843

[ Parent | on HN ]

Re:Apple M3 Ultra by whimsicalism on Wednesday March 05, 2025 @05:50PM (#43273857)

sure on consumer GPUs but that is not what is constraining the model inference in most actual industry setups. technically even then, you are CPU-GPU memory bandwidth bound more than just GPU memory, although that is maybe splitting hairs

[ Parent | on HN ]

Re:Apple M3 Ultra by cubefox on Wednesday March 05, 2025 @06:27PM (#43274228)

Why are industry setups considered actual while others are not?

[ Parent | on HN ]