Snowflake Arctic Instruct (128x3B MoE), largest open source model

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by kristianp on Wednesday April 24, 2024 @10:55PM (#40152836)

1 H100 is only 80GB of HBM. I guess you mean a server with 4xH100 is 1 node?

[ Parent | on HN ]

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by karmasimida on Thursday April 25, 2024 @02:41AM (#40154230)

this is essentially 400b params. With FP8, comparing to Grok'3 320B model, which requires 320GB VRam in int4, I think what the OP meant is actually 8 H100.

Which is ... a lot to say the least.

And all optimization is for latency, not throughput, because with 8 H100, you can easily hosted 4 replicas of 70B.

[ Parent | on HN ]

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by kristianp on Thursday April 25, 2024 @03:41AM (#40154617)

Thanks for the correction, there are indeed 8x nodes. https://developer.nvidia.com/blog/introducing-nvidia-hgx-h10...

[ Parent | on HN ]