News for hackers. Stuff that matters.

Show HN: Gall – Inkling 2 Mini on 24GB M2, frees RAM between messages

https://github.com/ltrgoddard/gall

Thinking Machines Lab gives 32GB as the minimum RAM requirement for Inkling 2 Mini on Mac, but with the Q3 quant I felt it must be possible to get somewhere reasonable on my 24GB M2 MacBook Air.

Llama.cpp wasn't cutting it even with patches, so I spent this week's Fable allowance (with a sprinkling of Astra) on this custom inference engine, and took the opportunity to add some optimisations and ergonomics for running on a memory-constrained laptop:

- strictly one model loaded at a time, requests queued; - KV cache gets saved and model unloaded after every response (or configurable delay), so RAM stays at 50MB between calls, but reload is fast; - menu bar app shows status and icon changes when prefilling/generating.

So far I've been using it with hax [1] with my normal system prompt + tool list, which runs to about 2,800 tokens, and getting near-instant responses in new sessions due to the cache-and-reload. Works great as a local agent setup for infrequent messages too, e.g. via a Telegram bridge.

I'm sure this vibecoded engine and the model itself will be superseded in time, but for now it's by far the best local model setup I've ever managed on this machine, so wanted to share for others in the same boat. Please do hack away at it.

1. https://github.com/OleksandrChekhovskyi/hax

( 1 comment | on HN )
Re:Show HN: Gall – Inkling 2 Mini on 24GB M2, frees RAM betw... by omegaSigma95 on Sunday November 22, 2026 @05:14PM (#50604504)
Dodged 32GB of ram with a week of fable credits, classic engineer move, love it