News for hackers. Stuff that matters.

A comment on Snowflake Arctic Instruct (128x3B MoE), largest open source model. Parent comment

Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour... by karmasimida on Wednesday April 24, 2024 @09:36PM (#40152292)
That is my reading too, if you consider latency as the utmost inference metric, then you need all models in memory all the time.

What is you guys 70B configuration, do you guys try TP=8 for the 70B model for a fair comparison?

[ Parent | on HN ]

There are no comments on this comment yet.