News for hackers. Stuff that matters.
Re:OpenAI Major Outage by motoboi on Wednesday January 25, 2023 @09:59PM (#34527229)
Something like 1,5Tb memory to run this model in inference mode.
Re:OpenAI Major Outage by antonvs on Wednesday January 25, 2023 @11:18PM (#34527632)
Is that really all? We regularly run multi TB memory clusters for big data processing and ML. I imagined it would be much bigger than that.

To put that in perspective, 24x 64 GB nodes is 1.5 TB.

Re:OpenAI Major Outage by albntomat0 on Wednesday January 25, 2023 @11:55PM (#34527811)
My understanding is that all the memory has to be GPU memory, with proper interconnects. Still not that crazy, all things considered
Re:OpenAI Major Outage by lossolo on Wednesday January 25, 2023 @11:56PM (#34527816)
> 24x 64 GB nodes is 1.5 TB

Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.

Re:OpenAI Major Outage by lossolo on Wednesday January 25, 2023 @11:58PM (#34527827)
You meant 700 GB? 32 bit precision in 175B model is around ~700GB + overhead or around 350 GB if they use half-precision.
Re:OpenAI Major Outage by motoboi on Thursday January 26, 2023 @08:56PM (#34541531)
I was wrong. It seems they use fall precision. So 350gb.

What is amazing, human language (languages?) and knowledge encoded in so little space.