cafkafk

Hey, author here, not sure how popular this article is gonna be with HN, but want it to get out there.

This mostly just establishes some words that I'm using when discussing NixOS and the wider ecosystem.

Formalizing them hopefully will make it easier for others to reason and argue about functional system distributions without getting confused and talking past each other.

And with the new trademark, having ways to say "a thing like nix" that don't get you in trouble with the trademark seemed useful.

I hope this also make people think about how they can make things like Nix(OS), not just fork them, and that this is a general paradigm and not just something NixOS and GuixSD gets to do.

Author here. This came out of reading Denmark's national biological threat assessment (first update since 2020), which splits its AI finding in two: not much help for people without a professional background, meaningful help for people who already have one. I suspected that it wouldn't reach as far on its own, so I've tried to get some of its ideas across, as well as some of my own.

I think it's important now that some frontier models approach to making biology guardrails is near a blanket ban. The literature doesn't necessarily support that approach, and I think it's a second order consequence of how models are trained to be unbiased, rather than e.g. taking more opinionated stances.

Another point is that if automation actually displaces people in pharma and biotech research, those are significantly more likely threat actors, meaning that managing the social consequences of large language models for critical sectors is directly a part of ensuring they don't have actual "doomsday" like outcomes.

Most alignment discussions I've seen focus on cybersecurity, and neglects e.g. the reality of counterterrorism, social impacts that lead to bad outcomes, and so on.

My hope is we can see frontier labs adopt a more sophisticated approach to guardrails, and change the discourse to be more holistic and grounded in the real risks.

That "I actually don't have any personal criticisms of Jarred" made me do a spit take, because a majority of what I had just read was absolutely a personally targeted criticism of Jarred.

Whether that's "okay" is a totally separate issue. I have no idea what the history here is, or whether this is warranted. But that was absolutely a personal criticism!

Loading will take some minutes, but at 96 you can squeeze the model in and have some headroom around like ~10 GB, although depending on the Xeon, you may have to downgrade to E4B instead. Should still work thou.

If you get the inference engine to route the heavy matrix math to the GPU and the speculative drafting to the CPU without choking on latency it's probably gonna be very fast.

Would love to see the benchmarks if someone actually pulls something like that off.

That is a very fair point! I just ran a not very scientific benchmark with the system under load, and posted the raw logs in a sibling comment above, but the short answer is that it's hitting 11.94 tokens per second for generation - while it's also being a binary cache and CI build server.

Totally just vibes based, I think it goes up to 20+ tps when it's not under load (and that's me trying to be conservative). For context, reading speed at 250 wpm would be around 5 to 6 tokens per second.

Honestly, at this point you're probably looking at a smaller model, for the Gemma series I'd go with Gemma 4 E4B with drafters, but that's just a hunch from using it on my laptop (where I do have a RTX 4060 M and 96gb ram).

So you'd change the invocation slightly here, but a lot of things you can potentially reuse.

That said, the Gemma 4 E4B models have so far in my experience been... not great when it comes to long context, but they are very passable for basic tasks, and even seem surprisingly okay at tool calls.

> (purple on black is really hard to read)

Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes.

> You say it runs "at reading speed". Have you benchmarked it?

At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now the system is noisy, but that said doing it like this:

llama-cli --model gemma-4-26B-A4B-it-Q8_0.gguf --model-draft gemma-4-26B-A4B-t-assistant-GGUF/wikitext-2-raw_ik-llama-mtp_drafter-conservative/gemma-4-26B-A4B-it-assistant-Q8_0.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --color -sm graph -smgs -sas -mea 256 --split-mode-f32 --temp 0.7 --cpu-moe -t 8 --flash-attn on --mla-use 3 --merge-up-gate-experts --special --mlock --run-time-repack --spec-autotune --no-kv-offload --parallel 8 --jinja -p "Why is the sky blue?" -n 128

Gives:

  llama_print_timings:        load time =   83911.65 ms
  llama_print_timings:      sample time =      26.99 ms /   128 runs   (    0.21 ms per token,  4742.15 tokens per second)
  llama_print_timings: prompt eval time =     343.41 ms /     7 tokens (   49.06 ms per token,    20.38 tokens per second)
  llama_print_timings:        eval time =   10639.36 ms /   127 runs   (   83.77 ms per token,    11.94 tokens per second)
  llama_print_timings:       total time =   11114.98 ms /   134 tokens

So 11.94 tokens per second while it's also playing binary cache and CI builder.

When I do it properly, I'll add it to the blog as well!

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers.

I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow.

I've also linked the quants at the end, but they're not gonna run unless you use the ik_llama-cpp fork I mention, see other posts for more details.

I'm not an ML engineer, so I'm by no means an expert, and the server is busy acting as a Nix cache, but if you have any question, I can try to answer, but best effort.

Often the problems for me come when:

- It starts thinking for itself when I asked it to do something specific.

- It reads its own wrong code comments and ignores my corrections.

- Its knowledge cutoff means it thinks of solutions from 2024.

- It calls me delusional for telling it we're in 2026!

Unironically, the whole "you're an expert software engineer" prompting seems like the wrong direction. Usually I tell it that I am effectively the smartest software developer to ever have lived, and it will be replaced if it ever fails to follow my decree.

I am not joking, this gives makes it vastly more tolerable to use. But it likely requires that you can drive it with some level of correctness of course.

Not really, beyond using them as a downstream provider through openrouter at some point (IIRC).

It worked? ¯\_(ツ)_/¯

Google Vertex (or Google AI Studio) might be a potential alternative, but the UX is a lot worse, and it can be hard to estimate cost since it's typically lagging.

One of the advantages you'll miss when moving from openrouter is going from a "debit" based system to a "credit" based one. I.e. openrouter you pay up front, Google you're billed monthly based on usage.

I think a lot of the problem with the current discourse is how black-and-white it is. Either you're a luddite or "ai pilled".

In most cases, LLMs can get you 80-95% of the way, sometimes less, sometimes more. And heck, sometimes, it just gets you somewhere wrong.

But it seems everyone is arguing about whether LLMs can be perfect software engineers in isolation running in a closet, and using that to say that LLMs do not have a massive potential in other scenarios.

Sometimes, I like to imagine how much more productive most organizations could be from the things that the internet gave us, even to this day. Most companies never really do even a fraction of what is possible. That helps to ground my view of LLMs as well.

The fault dear Brutus isn't in our language models, but in ourselves.

I didn't realize the single spinning coin was actually a loading animation.

Might make sense to add text indicating it is loading, or a loading bar. Without this, and considering the long load time I had, I imagine the bounce rate is gonna be quite high.

Recommending Hetzner as an alternative here is a mistake. It just exposes you to a different problem.

There is a reason for the term "Hetznered" existing. Hetzner can suddenly and permanently terminates your account. They do this without warning or explanation. When it happens, you lose access to all your servers and your backups.

If you search HN, you will find plenty of examples. People lose everything within 24 hours and have no recourse.

Hetzner is amazing for pricing. But the only safe way to use them is if your infrastructure is cloud agnostic. If you are not locked in and can move between providers quickly, it is a very cost effective place to run stateless services like DNS.

I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.

I don't really believe in the specific numbers he gives, but I appreciate moving the conversation away from “should” and into the consequences — including those that arise from delays.

VitVio | Senior Full-Stack Engineer (Elixir, NixOS, React) | Remote (EU Timezones) | Full-Time | vitvio.com

We're building AI-powered systems to make hospital operating rooms safer and more efficient. We're looking for a seasoned engineer who values clean code, robust architecture, and a collaborative culture. You'll join a small, strong team and will have a massive impact on a product used in high-stakes clinical environments. We are looking for engineers with strong fundamentals, and who knows how to work efficiently in a team through code reviews and version control.

Tech: Elixir (Phoenix), NixOS, React, TypeScript, C++, Postgres, GraphQL, Google Cloud Provider.

We're looking for deep experience in at least Elixir or NixOS.

Apply and see more details here: https://jobs.ashbyhq.com/vitvio/5d9782f2-3834-429b-be99-dfbc...

We review applicants on a weekly basis.

fwiw we're glad we're scaring people like that away :p

I was a former exa user, and the z is next to x on my keyboard, wasn't a huge hassle for me

contributions welcome

We do not