Run the strongest open-source LLM model: Llama3 70B with just a single 4GB GPU

Re:Run the strongest open-source LLM model: Llama3 70B with ... by logicallee on Wednesday June 26, 2024 @04:30PM (#40804283)

how much disk space did it use?

[ Parent | on HN ]

Re:Run the strongest open-source LLM model: Llama3 70B with ... by p1esk on Wednesday June 26, 2024 @04:53PM (#40804481)

I didn’t check, but iirc it was an fp16 model checkpoint which we converted to int8 for inference, so I assume 140GB?

[ Parent | on HN ]

Re:Run the strongest open-source LLM model: Llama3 70B with ... by logicallee on Thursday June 27, 2024 @03:34AM (#40808272)

Thanks, I finished downloading it (which took many hours) onto an external hard drive (by adding a HF_HOME environmental variable for where to store that cache). Its size was 262 GB.

[ Parent | on HN ]