MozerJ

TTFT is around 180ms, prefill speed is ~2.7k tok/sec. Yes, both prefill and decode slows down as concurrency increases (it's not linear). This 255 tok/s is at conc = 1.

It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls and other syntax related tasks.

This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.

That is true, the lack of good benchmarks will be with us for a while I think. Ternary is not a good quant algo, but because it's so drastic, it allows people to use enormous models on worse hardware, or just increase throughput speed. In the meanwhile, there's research going into how this quantisation can be executed while correcting for the loss. In this case we're not just quantising but also post training the model, and we can "trade" some capabilities to others (ex. post-train the experts for math but lose on biology or whatever).

With that said, I find nvfp4 to be far superior. I think right now all you can do is try out the models, maybe run your own benchmarks if you have.

Lilith-zero is runtime enforcement, not pre-connection trust. It focuses on what happens once traffic flows (for example are tool calls authorised, are taint policies satisfied).

On server identity: the middleware spawns the upstream binary directly, so identity is established at launch, not inferred from a URL

The signed manifest idea is good, the security core already HMAC-validates session tokens on every tool request, extending that to a pre-handshake attestation is a next step yes.

Lilith-zero sits between the agent and the tool server as a transparent proxy, meaning that the agent speaks to the middleware over stdio, and the middleware manages the upstream server process directly (spawning, supervising, and killing it). Your agent config just points at lilith-zero instead of the server binary.

On latency: the measured overhead is ~247 ns for codec framing and ~660 ns per policy rule evaluation. End-to-end p50 RPC overhead on a release build is under 0.5ms (M4 apple silicon). It's implemented in rust, so it's designed to be invisible in practice. This is yet to be perfected tho, in some complicated policy cases, it could take a couple of µs per validation, and with agent swarms this may add up.