This is similar with the notion of time. A typical perception is that only now exists. Yet according to physics there is no now. All our physical models based on experience imply that the universe is 4-dimensional static something. There is no now and all points across the time dimension have same properties just as points across space.
One needs metaphysics to try to explain this discrepancy between perception and very successful physical models.
I'm not sure what you mean by this so I'm also not sure how to address it, but it does seem reasonable to assume free will simply does not exist exactly because phenomena is either deterministic or stochastic, not some third option which would allow free will. This view is informed by physics.
> There is no now and all points across the time dimension have same properties just as points across space.
This is a much more interesting problem and one that has kept me up many times.
As for the problem of now, for me it is similar to the problem of free will. Starting from Parmenides and Buddha one way to resolve this was to declare that the perception of now and movement is an illusion similar to the notion of free will. And as with free will, that will be compatible with physical models and the opposite cannot be expressed within physical models.
> But this has nothing to do with the determinism of physical models.
It seems rather confusing to state it has nothing to do with determinism of physical models. More accurately, it does have to do with determinism of physical models if you assume a physicalist perspective, but it might not, though then you have to resort to much more involved and comprehensive models of what consciousness and will are (like changing both the future and past).