macwhisperer

lmao *qwen3.8-27b runs in the background*

honestly boggles my mind that actual companies put load-bearing (lol) work behind 3rd party api's like claude or gpt.. also its really funny because some of these companies fired actual humans to put said 3rd party api to work in place of humans... like what happens when ur boy claude stops showing up for work huh!?

super dope!! the future of musical instruments may be web-based.. exciting times

4-bit quant sits at 15.72GB

yeah this model is chefs kiss..

running an untouched, vanilla 4-bit version (Q4_0) I baked myself today (benched it against Q4_K_M (16gb) and IQ3_M (12gb), Q4_0 (15gb) is king)...

this model--

1: over 60% faster than qwen 3.6 version of the same dense 27b model, same engine setup (don't ask how, im not sure either)

2: has better reasoning quality, less "loopy" with its thinking patterns.. most certainly the smartest model on my roster currently

3: has the longest task horizon ive ever experienced (locally or otherwise)...I sent it a bunch of compressed ideas for an app, it sent me back the largest python app ive ever seen in one single ai response pass (80kb text file)

thanks qwen!! hoping to see the full model range get released...

pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...

I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak

big week for open models... seems like companies are noticing the 26-35b sweet spot... though I think a 12b-a1b-MoE model would be helpful for the 16gb folks

I feel like all the ppl complaining itt don't even run open weight models.. wahh billionaire bad is true, but you are missing the forrest for the trees.. they are literally bending the knee and ur mad about it? it literally means local models won.

yeah you realize you can turn reasoning off on the newer the models right? also try my version of Gemma-12b https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense or try my qwen 3.5-9b

Basically, he proved that *information is power.* If you don't know which way to go (the subgradient), you're gonna be calculating forever!

also for those with only 16gb-- try this model https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense its exceptional!

hi guys... I run specialized quants on my 24gb air.. (I specialize in 3-bit quants that punch above their weight).. try out my version of 3.6-27b I think you be impressed https://huggingface.co/macwhisperer/Qwen3.6-27B-SuperDense

qwen3 1.7b- q4_k_m is your best best for that size

literally ask cloud ai like the free gemini or chatgpt.. they could make you an expert on the subject overnight..

I code with like a slew of 20+ custom baked models of all sizes, in various fully custom multi-model harnesses that use different bindings...

the harnesses themselves are just as important as the models...different harnesses give different responses with the same prompt, same model...

if you have the 20/mnth claude sub or codex, you really should be using that to build a good local harness for yourself... claude won't be 20$ forever

build the stack first! when you get that new comp with massive ram, youre already set, just run a larger model!

big cloud models are incredibly good at building and teaching about local ai!

have fun in the rabbit hole!

if you are memory constrained like me, check out my custom models https://huggingface.co/macwhisperer

ai is like the first technology with a conversational service manual inside it..

you should be foaming at the mouth to use claude or codex to make a custom harness, just for your own personal use with local models...

super inspiring! thanks for sharing!

retro-inspired fully custom, swiss army knife style notepad --

https://convert.neocities.org

check out a custom 4-bit quant I made today

https://huggingface.co/macwhisperer/Gemma4-12B-SuperDense

should run perfect for 12-16gb with maybe 10-20k context

seems intelligent enough that I would recommend this as a daily driver for friends who just want a local ai that can do most things relatively quickly (getting 10 tps on my m2 air)

the HITL (human in the loop) is basically the single point...AI is a mirror..

it only "exists" when you talk to it.. much like your reflection in the mirror is only there when you're in view.

models can never be self-improving because it can never have "self". it can only mirror the appearance of self.

what's actually happening is "symbiotic group improvement".

our brains are resonant.. for those of use who are brilliant, getting leverage with ai just means that our innovative ideas become louder and more physically real every day.

eventually everything worth building will be built for free and made readily available.. no more "profiteering"

its Jevons paradox "efficiency breakthrough -> effort reduces -> growth potential rises -> transformative gains happen"...

some of us are in the "transformative phase"..

others haven't seen the "breakthrough moment" yet, but they will soon.

this is cool thanks for making it!

cool! what stack are you using for the multiplayer?

can't we just freaking share things?

ai models are a crystallization of human effort (available for free on huggingface)..

why not use it?

AI is like the UBI of intelligence, stop leaving free money on the table by refusing to use it locally ( you can run it on a laptop CPU)

this is really cool congrats!

good, the point is that now you have free mental bandwidth to use on building something that truly interests you using AI to help actualize your goal. build something cool for your kiddos idk?

at this point I trust software companies less and less..

being able to build the stack and create bespoke solutions with llms's is incredible..

idk why people get mad about vibe-coding. if ur little brother can make a Spotify clone with Claude in a day, shouldn't that mean that you as a dev should be able to create something 100x better that makes Spotify obsolete?

good ideas / feasible novel architecture design will be the only thing valuable..

I run the latest 20b-30b models on a MacBook Air... running inference with an MoE (25 tps) for like 2 hours is like 10% battery.. (look me up on huggingface to download my models)

also you gotta realize frontier models have massive "system prompts" that clog up the context window with garbage.

being able to write your own system prompts gives you a MASSIVE edge..

ai is exciting because it shows us what really matters...