If you can do it offline, batching solves this. We used a test dataset from CFPB and at n=20, it was 1.6x faster and 1.2x more expensive with no statistically meaningful accuracy dropoff. Did not tune for batch size, but it's possible that we could get to 30 and see better perf.

That's not one if, it's two big ifs. If you can to it offline. If you can batch. I wouldn't say that's all that common in the use cases Jev sells itself for. You often want the answer now, not "averaged across 20 items in a batch that took 20 times longer".

It's fair to note it, but I wouldn't say "batching solves this" so much as "batching can solve time insensitive use cases".

No shit its cheaper and faster, thats the case with every small model.

It's a trade off between wanting something dumber but fast and cheap, or something slower and more expensive that is vastly more capable and smarter.

But the thing is that across a lot of classification benchmarks, Jev is actually faster, cheaper, and smarter than Luna (at least without reasoning). For example, look at this one from another user in this thread [1]

[1] https://hard-decisions.anth.us/models/

The reason the comparison is being made is because Jev and Luna 6 have similar levels of accuracy...