Serving AI from the Basement – 192GB of VRAM Setup

Re:Serving AI from the Basement – 192GB of VRAM Setup by angoragoats on Sunday September 08, 2024 @02:53PM (#41482282)

Yep. The 8x3090 setup should be at least 10 times faster than the Mac.

[ Parent | on HN ]

Re:Serving AI from the Basement – 192GB of VRAM Setup by wkat4242 on Sunday September 08, 2024 @03:22PM (#41482490)

Yes though for inference memory bandwidth is more important than tflops. Not sure how Apple compares in that regard. The OP does train models too though which is more compute heavy.

[ Parent | on HN ]

Re:Serving AI from the Basement – 192GB of VRAM Setup by angoragoats on Sunday September 08, 2024 @03:36PM (#41482582)

Each of the 3090s has something like 15% higher memory bandwidth than the whole M2 Ultra system does.

[ Parent | on HN ]