News for hackers. Stuff that matters.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by the_mitsuhiko on Friday January 31, 2025 @07:19AM (#42886922)
Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind.

However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

Re:Llama.cpp supports Vulkan. why doesn't Ollama? by 7thpower on Friday January 31, 2025 @07:28AM (#42886985)
Can you please explain why you think they may be operating in bad faith?
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by diggan on Friday January 31, 2025 @08:20AM (#42887364)
Not parent, but same feeling.

First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library.

Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture.

Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "Misinformation" and rewrites your messages. You can ask a honest question, it gets deleted and you get blocked for a day.

Just today I asked why the R1 models they're shipping that are the distilled ones, doesn't have "distilled" in the name, or even any way of knowing which tag is which model, and got the answer "if you don't like how things are done on Ollama, you can run your own object registry" which doesn't exactly inspire confidence.

Another thing I noticed after a while is that there are bunch of people with zero knowledge of terminals that want to run Ollama, even though Ollama is a project for developers (since you do need to know how to run a terminal). Just making the messaging clearer would help a lot in this regarding, but somehow the Ollama team thinks thats gatekeeping and it's better to teach people basic terminal operations.

Re:Llama.cpp supports Vulkan. why doesn't Ollama? by davely on Friday January 31, 2025 @12:12PM (#42889447)
For what it's worth, HuggingFace provides documentation on how you can run any GGUF model inside Ollama[0]. You're not locked into their closed library or have to wait for them to add new models.

Granted, they could be a lot more helpful in providing information on how you do this. But this feature exists, at least.

[0] https://huggingface.co/docs/hub/en/ollama

Re:Llama.cpp supports Vulkan. why doesn't Ollama? by diggan on Friday January 31, 2025 @11:13AM (#42888943)
Ollama team's response (verbatim) when asking what they think of the comments about Ollama in this HN submission: "Who cares? It's the internet... everybody has an opinion... and they're usually bad". Not exactly the response you'd expect from people who should ideally learn from what others think (correct or not) about your project.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by 7thpower on Friday January 31, 2025 @02:39PM (#42891019)
This is helpful, thank you.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by blixt on Friday January 31, 2025 @07:41AM (#42887093)
LM Studio has been around for a long time and does a lot of similar things but with a more UI-based approach. I used to use it before Ollama, and seems it's still going strong. https://lmstudio.ai/
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by buyucu on Friday January 31, 2025 @07:50AM (#42887150)
isn't lm stuido closed source?
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by prabir on Friday January 31, 2025 @07:27AM (#42886980)
There is https://cortex.so/ that I’m looking forward too.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by adastra22 on Friday January 31, 2025 @09:02AM (#42887745)
Hey thanks, I didn't know about cortex and this looks perfect.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by buyucu on Friday January 31, 2025 @07:23AM (#42886949)
I totally agree that ollama needs competition. They have been doing very sketchy things lately. I wish llama.cpp had an alternative wrapper client like ollama.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by Liquix on Friday January 31, 2025 @07:35AM (#42887048)
agreed. but what's wrong with Jan? does ollama utilize resources/run models more efficiently under the hood? (sorry for the naivete)
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by Deathmax on Friday January 31, 2025 @08:31AM (#42887447)
The most recent one of the top of my head is their horrendous aliasing of DeepSeek R1 on their model hub, misleading users into thinking they are running the full model but really anything but the 671b alias is one of the distilled models. This has already led to lots of people claiming that they are running R1 locally when they are not.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by TeMPOraL on Friday January 31, 2025 @09:13AM (#42887875)
The whole DeepSeek-R1 situation gets extra confusing because:

- The distilled models are also provided by DeepSeek;

- There's also dynamic quants of (non-distilled) R1 - see [0]. Those, as I understand it, are more "real R1" than the distilled models, and you can get as low as ~140GB file size with the 1.58-bit quant.

I actually managed to get the 1.58-bit dynamic quant running on my personal PC, with 32GB RAM, at about 0.11 tokens per second. That is, roughly six tokens per minute. That was with llama.cpp via LM Studio; using Vulkan for GPU offload (up to 4 layers for my RTX 4070 Ti with 12GB VRAM :/) actually slowed things down relative to running purely on the CPU, but either way, it's too slow to be useful with such specs.

--

[0] - https://unsloth.ai/blog/deepseekr1-dynamic

Re:Llama.cpp supports Vulkan. why doesn't Ollama? by zozbot234 on Friday January 31, 2025 @09:17AM (#42887908)
> it's too slow to be useful with such specs.

Only if you insist on realtime output: if you're OK with posting your question to the model and letting it run overnight (or, for some shorter questions, over your lunch break) it's great. I believe that this use case can fit local-AI especially well.

Re:Llama.cpp supports Vulkan. why doesn't Ollama? by adastra22 on Friday January 31, 2025 @08:59AM (#42887728)
I'm not sure that's fair, given that the distilled models are almost as good. Do you really think Deepseek's web interface is giving you access to 671b? They're going to be running distilled models there too.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by Deathmax on Friday January 31, 2025 @09:39AM (#42888111)
It's simple enough to test the tokenizer to determine the base model in use (DeepSeek V3, or a Llama 3/Qwen 2.5 distill).

Using the text "സ്മാർട്ട്", Qwen 2.5 tokenizes as 10 tokens, Llama 3 as 13, and DeepSeek V3 as 8.

Using DeepSeek's chat frontend, both DeepSeek V3 and R1 returns the following response (SSE events edited for brevity):

  {"content":"സ","type":"text"},"chunk_token_usage":1
  {"content":"്മ","type":"text"},"chunk_token_usage":2
  {"content":"ാ","type":"text"},"chunk_token_usage":1
  {"content":"ർ","type":"text"},"chunk_token_usage":1
  {"content":"ട","type":"text"},"chunk_token_usage":1
  {"content":"്ട","type":"text"},"chunk_token_usage":1
  {"content":"്","type":"text"},"chunk_token_usage":1
which totals to 8, as expected for DeepSeek V3's tokenizer.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by adastra22 on Friday January 31, 2025 @10:49AM (#42888779)
I’m not sure I understand what this comment is responding to. Wouldn’t a distilled Deepseek still use the same tokenizer? I’m not claiming they are using llama in their backend. I’m just saying they are likely using a lower-parameter model too.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by zozbot234 on Friday January 31, 2025 @11:03AM (#42888865)
The small models that have been published as part of the DeepSeek release are not a "distilled DeepSeek", they're fine-tuned varieties of Llama and Qwen. DeepSeek may have smaller models internally that are not Llama- or Qwen-based but if so they haven't released them.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by adastra22 on Friday January 31, 2025 @12:32PM (#42889651)
Thank you. I’m still learning as I’m sure everyone else is, and that’s a distinction I wasn’t aware of. (I assumed “distilled” meant a compressed parameter size, not necessarily the use of another model in its construction.)
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by zozbot234 on Friday January 31, 2025 @09:03AM (#42887763)
Given that the 671B model is reportedly MoE-based, it definitely could be powering the web interface and API. MoE slashes the per-inference compute cost - and when serving the model for multiple users you only have to host a single copy of the model params in memory, so the bulk doesn't hurt you as much.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by adastra22 on Friday January 31, 2025 @09:27AM (#42887998)
They can still run a lot more users on the same number of GPUs (and they don't have a lot) using distilled models.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by imtringued on Friday January 31, 2025 @10:07AM (#42888381)
Ollama doesn't really need competition. Llama.cpp just needs a few usability updates to the gguf format so that you can specify a hugging face repository like you can do in vLLM already.
Re:Llama.cpp supports Vulkan. why doesn't Ollama? by justinmayer on Friday January 31, 2025 @05:00PM (#42892690)
Benefiting users is definitely not Ollama’s first priority, as seen when this pull request was summarily closed: https://github.com/jmorganca/ollama/pull/395

Those README changes only served to provide greater transparency to would-be users.

Ulterior motives, indeed.