Re:Devstral
by throwaway314155 on Thursday May 22, 2025 @12:20AM
(#44058748)
> For people without a 24GB RAM video card, I've got an 8GB RAM one running
What're you using for this? llama.cpp? Have a 12GB card (rtx 4070) i'd like to try it on.
Re:Devstral
by johnQdeveloper on Thursday May 22, 2025 @12:23AM
(#44058765)
https://ollama.com/library/devstral
I believe its just a HTTP wrapper and terminal wrapper around llama.cpp with some modifications/fork.
Re:Devstral
by throwaway314155 on Thursday May 22, 2025 @12:38AM
(#44058830)
Does ollama have support for cpu offloading?
Re:Devstral
by johnQdeveloper on Thursday May 22, 2025 @01:49AM
(#44059120)
> Does ollama have support for cpu offloading?
https://www.reddit.com/r/ollama/comments/1df757o/high_cost_o...
https://github.com/ollama/ollama/issues/8291
Yes.
A perfect blend of LMGTFY and helpfulness. :)
Re:Devstral
by johnQdeveloper on Thursday May 22, 2025 @04:29AM
(#44059984)
lol. I try not to be a total asshole, it sometime even works! :)
Good luck to you mate with your life :)
total duration: 35.016288581s load duration: 21.790458ms prompt eval count: 1244 token(s) prompt eval duration: 1.042544115s prompt eval rate: 1193.23 tokens/s eval count: 213 token(s) eval duration: 33.94778571s eval rate: 6.27 tokens/s
total duration: 4m44.951335984s load duration: 20.528603ms prompt eval count: 1502 token(s) prompt eval duration: 773.712908ms prompt eval rate: 1941.29 tokens/s eval count: 1644 token(s) eval duration: 4m44.137923862s eval rate: 5.79 tokens/s
Compared to an API call that finishes in about 20% of the time it feels a bit slow without the recommended graphics card and what not is all I'm saying.
In terms of benchmarks, it seems unusually well tuned for the model size but I suspect its just a case of gaming the measurement by testing against it as part of the development of the model which is not bad in and of itself since I suspect every LLM who is in this space marketed to IT folks does the same thing tbh so its objective enough given that as a rough gauge of "Is this usable?" without heavy time expense testing it.