AI engineers claim new algorithm reduces AI power consumption by 95%

Re:AI engineers claim new algorithm reduces AI power consump... by dragonwriter on Monday October 21, 2024 @03:04PM (#41907290)

OTOH, doesn't it also mean that (given appropriate software framework support) iGPUs with less processing capacity and slower-but-more RAM available (because system RAM is comparatively cheap and plentiful compared to VRAM) without swapping anything are more competitive against consumer dGPUs with fast-but-small RAM for both inference and training with larger models?

[ Parent | on HN ]

Re:AI engineers claim new algorithm reduces AI power consump... by Sohcahtoa82 on Monday October 21, 2024 @05:02PM (#41908501)

System memory isn't that fast, either. Even with DDR5-8400, the fastest memory you can get right now, you're only looking at a memory transfer speed of 67.2 GB/s, barely faster than the PCI-E bus. So even if you could store that entire 70B model in RAM, you're still getting just under 1 token/sec, and that's assuming your CPU doesn't become a bottleneck.

Your best bet would likely be a laptop that has integrated system RAM with VRAM, but I don't think any of those offer enough RAM to store an entire 70B model. A 7B parameter model would work fine, but you could do those on a consumer-grade GPU anyways.

[ Parent | on HN ]

Re:AI engineers claim new algorithm reduces AI power consump... by DeveloperErrata on Monday October 21, 2024 @08:35PM (#41910031)

Macbook Pros with M3 & integrated RAM & VRAM can do 70B models :)

[ Parent | on HN ]