News for hackers. Stuff that matters.
Re:Mixtral 8x22B by paradite on Wednesday April 17, 2024 @11:07AM (#40065763)
And how much RAM do you need to run Mixtral 8*22B? Probably not enough on a personal laptop.
Re:Mixtral 8x22B by user_7832 on Wednesday April 17, 2024 @11:13AM (#40065857)
Generally about ~1gb ram per billion parameters. I've run a 30b model (vicuna) on my 32gb laptop (but it was slow).
Re:Mixtral 8x22B by Lacerda69 on Wednesday April 17, 2024 @11:12AM (#40065839)
I run it fine on my 64gb RAM beast.
Re:Mixtral 8x22B by coder543 on Wednesday April 17, 2024 @11:16AM (#40065887)
At what quantization? 4-bit is 80GB. Less than 4-bit is rarely good enough at this point.
Re:Mixtral 8x22B by apexalpha on Wednesday April 17, 2024 @11:18AM (#40065927)
Is that normal ram of GPU ram?
Re:Mixtral 8x22B by samus on Wednesday April 17, 2024 @04:22PM (#40069671)
64GB is not GPU RAM, but system RAM. Consumer GPUs have 24GB at most, those with good value/price have way less. Current generation workstation GPUs are unaffordable; used can be found on ebay for a reasonable price, but they are quite slow. DDR5 RAM might be a better investment.