Since it's CPU vs GPU, it's not apples-to-apples. But in terms of wall-clock time, one inference pass is surely faster than 17 seconds. Plus there's the proof generation step, which the post describes as ~100x slower than generating the response itself.
21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.
Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.
The bigger factor is that (IIUC) ZKP-based verification is constant-time in the size of the output, so it definitely beats regen-based verification (as long as the constant is reasonable). And the ability to verify on consumer hardware instead of a beefy GPU is important too.
I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!
As a client, you're still paying for the compute even if it happens on the service provider's side. Running the same computation twice (perhaps by another provider) is 2x the cost; but if you ask the provider to make this proof, that increases the cost 70-95 times according to the article.
With that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap.
> The verifier runs on the client's CPU. It needs no GPU and only has access to a commitment to the weights. The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's checks.
As a client that sounds like it might be much cheaper to verify than reprocessing.