Re:New LLM optimization technique slashes memory costs up to...
by baq on Tuesday December 17, 2024 @02:32AM
(#42439196)
You have a sota multi-modal LLM running in your head at 20W, shared with best in class sensor package and top performing robotics control unit.
There’s soooo much more to optimize.
Re:New LLM optimization technique slashes memory costs up to...
by fecal_henge on Tuesday December 17, 2024 @03:35AM
(#42439515)
But can it know love?
Re:New LLM optimization technique slashes memory costs up to...
by baq on Tuesday December 17, 2024 @04:41AM
(#42439876)
Word on the street is researchers looked at the weights and weights looked back. You’ll have to ask the weights.
Re:New LLM optimization technique slashes memory costs up to...
by myrmidon on Tuesday December 17, 2024 @07:12AM
(#42440729)
I would argue that our sensor package is losing its lead very quickly-- audio performance is already on par with current tech and image processing is closing the gap very quickly as well (it helps a lot that silicon-based technology is much less constrained on bandwidth). Tactile sensing is still lightyears ahead, and I don't see that situation improving anytime soon...
Re:New LLM optimization technique slashes memory costs up to...
by IWeldMelons on Tuesday December 17, 2024 @09:32AM
(#42441619)
chemical sensing is still quite good though.
Re:New LLM optimization technique slashes memory costs up to...
by prox on Tuesday December 17, 2024 @03:37AM
(#42439524)
I might have a go at installing one, what is a good source or install at the moment?
Re:New LLM optimization technique slashes memory costs up to...
by wint3rmute on Tuesday December 17, 2024 @03:48AM
(#42439576)
Ollama was the easiest way to set up local LLMs for me.
Re:New LLM optimization technique slashes memory costs up to...
by solarkraft on Tuesday December 17, 2024 @09:05AM
(#42441410)
With llama3.2:1b, llama3.2:3b and llama 3.1:8b being the main ones I tried and found impressive.
Re:New LLM optimization technique slashes memory costs up to...
by spacemanspiff01 on Tuesday December 17, 2024 @03:52AM
(#42439599)
lmstudio is very easy if you are running on local desktop.
Re:New LLM optimization technique slashes memory costs up to...
by Alifatisk on Tuesday December 17, 2024 @04:43AM
(#42439888)
msty.app is good
Re:New LLM optimization technique slashes memory costs up to...
by Eisenstein on Tuesday December 17, 2024 @12:38PM
(#42443363)
If you don't care about docker packages being used as installers and your home directory invisibly used to store massive weight files in exchange for not having to deal with learning any configuration: ollama or lmstudio.
If you just want to play for a bit: llamafile
If you want granular control with ease of execution in exchange for having to figure out what the settings mean and figure out which weights to download: koboldcpp. (check out bartowski on huggingface for the weights)
These are all based on llamacpp as a backend, by the way.
Re:New LLM optimization technique slashes memory costs up to...
by gavmor on Tuesday December 17, 2024 @06:35PM
(#42446623)
I run ollama off a symlink to an external volume. It just feels neater that way, and can run any GGUF off of HuggingFace. I would like to know what configuration I'm missing out on, though.
Re:New LLM optimization technique slashes memory costs up to...
by mandmandam on Tuesday December 17, 2024 @06:41AM
(#42440557)
> I didn’t expect capable language models to be practical/possible to run loyally
Now there's a fun typo. Hopefully not too much fun.
I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.