News for hackers. Stuff that matters.
Re:Llamafile lets you distribute and run LLMs with a single ... by abrinz on Wednesday November 29, 2023 @05:55PM (#38466480)
I've been playing with various models in llama.cpp's GGUF format like this.

  git clone https://github.com/ggerganov/llama.cpp     

  cd llama.cpp

  make 

  # M2 Max - 16 GB RAM

  wget -P ./models https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF/resolve/main/openhermes-2.5-mistral-7b-16k.Q8_0.gguf
  
  ./server -m models/openhermes-2.5-mistral-7b-16k.Q8_0.gguf -c 16000 -ngl 32

  # M1 - 8 GB RAM 

  wget -P ./models https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF/resolve/main/openhermes-2.5-mistral-7b.Q4_K_M.gguf

  ./server -m models/openhermes-2.5-mistral-7b.Q4_K_M.gguf -c 2000 -ngl 32
Re:Llamafile lets you distribute and run LLMs with a single ... by m1thrandir on Wednesday November 29, 2023 @06:45PM (#38467069)