DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but 95% less expensive

Re:DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but... by jyap on Wednesday October 30, 2024 @04:35PM (#41999951)

This 236B model came out around September 6th.

DeepSeek-V2.5 is an upgraded version that combines DeepSeek-V2-Chat and DeepSeek-Coder-V2-Instruct.

From: https://huggingface.co/deepseek-ai/DeepSeek-V2.5

[ Parent | on HN ]

Re:DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but... by genpfault on Wednesday October 30, 2024 @04:43PM (#42000041)

> To utilize DeepSeek-V2.5 in BF16 format for inference, 80GB*8 GPUs are required.

[ Parent | on HN ]

Re:DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but... by risho on Wednesday October 30, 2024 @08:00PM (#42001988)

I wonder if the new mbp can run it at q4.

[ Parent | on HN ]

Re:DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but... by throwdbaaway on Wednesday October 30, 2024 @10:21PM (#42002812)

Using https://github.com/kvcache-ai/ktransformers/, an intel/amd laptop with 128GB RAM and 16GB VRAM can run the IQ4_XS quant and decode about 4-7 token/s, depending on RAM speed and context size.

Using llama.cpp, the decoding speed is about half of that.

Mac with 128GB RAM should be able to run the Q3 quant, with faster decoding speed but slower prefilling speed.

[ Parent | on HN ]

Re:DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but... by metadat on Thursday October 31, 2024 @12:44AM (#42003573)

What is "prefiling"?

[ Parent | on HN ]

Re:DeepSeek v2.5 – open-source LLM comparable to GPT-4o, but... by ikeashark on Tuesday November 12, 2024 @11:13AM (#42116762)

Assuming you already know what context in terms of LLMs, prefilling is the process of converting the current conversation into tokens and passing that into the LLM.

[ Parent | on HN ]