You already can with ollama

I'm not aware of any Debian family distro that packages it, but NixOS has at least ollama and llama-cpp in its repos. Honestly even if the more stable distributions did have these things packaged, I would hesitate to use the packaged versions because all of this stuff is still so quickly moving that you'd be on an old version and it would hurt.

Edit: Arch has ollama in official repos too. OpenSUSE has https://software.opensuse.org/package/ollama .

You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable

And everything runs way faster on cpu like that

I could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD

With this model:

https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF

Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram

Ollama is close...

Realistically, you probably want to wait until Vulkan support trickles out. That way, you aren't at the whim of the various evil hardware drivers (everybody's suck), and the AI can give you a disappointingly confused answer much faster than running the LLM on a CPU can.