Also, in this case, the game name is not “Game A” but something like “Deep Seek v4 Pro”, which they have previously chosen to use to describe Deep Seek v4 Pro, not Deep Seek v4.1 Flash.
I work for an education department that serves a chatbot for students, and model changes go through painstaking content safety reviews. I initially assumed it's just a bunch of bureaucratic paranoia. But every other model upgrade has a measurably different adherence to the existing system prompts about not talking to the kids about sex and drugs and mental health issues.
I'm not being a d**, just saying, the problem you have is something that I have faced EXACTLY, and at least here it's not working until you host in house or remote but on raw hardware. Otherwise it keeps having subtle changes, and you will notice no LLM API providers has guarantees about these.
There's a whole spectrum between self-hosting open weight models and having a cloud provider swap models from under you
Should you self host a model if want to maximize predictability to the limit? Yes. Does that mean it's wrong for someone hitting a model on API to expect that it won't switch to a completely different model under the hood from one day to another? Probably not.
But even if we only asked for sonnet:latest, the last thing we'd expect is opus. Model names should be indicative of breaking changes, and change management doesn't just go in the bin because of non-deterministic tools.
Just the risk of such a thing means regression testing every time you update the model, and you want to be able to run that testing on your schedule rather than having it forced on you.
This isn't true. Even Sol messes up JSON formatting for me on occasion.
Do not delude yourself into thinking these things are reliable. They are not.
Replacing a six-sided die for an eight-sided die also keeps rolls non-deterministic.
That doesn't mean it's fine to just replace the dice mid-game.
Also in principle it's similar to Anthropic downgrading.
Personally I use the basis that if I don't self host (I include remote host, but that I pay per hosting nor per model or api), it can change behavior without me asking. But they shouldn't, but it doesn't matter that's what they do.
Sort of like shooting a rifle - where the bullets hit is (to some order of magnitude, no philosophizing please) non-deterministic, but different very similar rifles will group differently and need to be appropriately adjusted to hit anything.
Yes - model hosts can do nasty things to you aside from changing the underlying model. That doesn't mean it's cool to have them change the model automatically.
Yes, it would be preferable to have complete control over your model serving, and no - not everyone is in a position to do that themselves.
That flavor profile is known -- it's typical behavioral distribution is somewhat understood (and, often, common failure modes addressed). If JSON breaks about 20% of the time, and that drops for 2% or blows up to 90%, it can drive all sorts of issues (not the least, costs for retries).
> those things are not deterministic
Determinism was an explicit goal of DeepSeek-V4. From their paper: https://arxiv.org/html/2606.19348v1#S3.SS3 > we implement end-to-end, bitwise batch-invariant, and deterministic kernels with minimal performance overhead
Of course, providers may not implement deterministic inference for various reasons, but it is possible.They think that sampling is an inherent part of Transformers.
Even on this site, it is regurgitated with confidence.
The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution.
But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.
But I agree with you. I have a dumb workflow that worked well with v4-flash-0731 and I suspect is directing to a newer model that now breaks it.
if you want deterministic returns, you should set the temperature to 0 to get the best possibility of deterministic.
E.g. if I've written a role playing character using a specific model I may want to pin the character to that model until I've been able to test the model being "better" doesn't affect the feel of the character before switching. That doesn't mean I need the character's responses to be completely deterministic, but that doesn't imply I'm fine with the character having a different quality or feel of response just because the new model is out.
It'd be nice if there was a more explicit way to signal in the request "I want what you think is best per dollar for this class of answer" vs "I want this model to answer".
If I were paying anthropic prices, I'd expect it, but Deepseek is a super scrappy upstart in comparison and intentionally arbitraging on price. I would never expect them to do that.
I'm not sure that anyone will mind running production workloads against an API that bills half as much for a chunk of the day.
If they were still at original price I'd get a couple of DGX Sparks myself to run Flash models at a decent quant/context combo.
Pick Auto (Deepseek v4 Flash Auto vs Deepseek v4.x Flash), and let the vendor decide. I think OpenRouter uses this method.
Please don't do this kind of thing. If a user has validated a workflow on V4 Pro, they might not want to suddenly start testing it in production on V4.1 Flash. Instead, keep V4 Pro around but deprecated for a defined period of time, then remove it.
At least as open weights models, it's possible to use something like Together.ai or OpenRouter to run the V4 Pro model as long as other providers keep it up.