News for hackers. Stuff that matters.
Re:Ask HN: Has anyone replaced Claude/GPT with a local model... by cpburns2009 on Wednesday June 17, 2026 @03:03PM (#48575168)
My LLM host is a Framework Desktop 128GB (Ryzen AI 395) with an RTX 5090 (Minisforum DEG1 dock, cheap Oculink card plugged into the PCIe 4.0 x4 slot). I run llama-server (b9296 compiled for CUDA) using a systemd service, and its configured as:

    # server-2.env
    LLAMA_ARGS="--host 0.0.0.0 --port 8082 --sleep-idle-seconds 1800"
    LLAMA_ARG_MODELS_MAX=1
    LLAMA_ARG_MODELS_PRESET=/usr/local/etc/llama.cpp/models-2.ini

    # models-2.ini
    [*]
    device = CUDA0
    fit = on
    cache-type-k = q8_0
    cache-type-v = q8_0

    [Qwen3.6 27B It Q6]
    model = /mnt/data1/llm/huggingface/hub/models--unsloth--Qwen3.6-27B-GGUF/snapshots/82d411acf4a06cfb8d9b073a5211bf410bfc29bf/Qwen3.6-27B-UD-Q6_K_XL.gguf
    alias = qwen3.6-27b-it:q6
    ctx-size = 200000
    temp = 0.6
    top-p = 0.8
    top-k = 20
    min-p = 0.0
    presence-penalty = 1.5
    repeat-penalty = 1.0
    chat-template-kwargs = {"enable_thinking":false}
I use the model from a sandboxed OpenCode using the @ai-sdk/openai-compatible plugin. And I'm using an SSH tunnel from home to work to expose llama-server to OpenCode.

There are no comments on this comment yet.