I'd go through the receiver channel tests in the manual and try to isolate the problem to the test set or IF box. Does S21 drift as well?
1. First suspect is always a bad calibration -- they tend to be unstable in addition to incorrect, but if you have anything with known(-ish) S parameters you can check for correctness too.
2. Second step is to put it in TDR mode and watch to see where the TDR changes. That's where your problem is.
3. A trip through the test set with a torque wrench is a good way to not just check the connections but also calibrate your intuition about the couplers/mixers, which should help interpret #2. You can loosen connectors in sequence and watch them spike on TDR to zero in on the actual problem.
As it happens, I was comparing my 8510C + 8517B to a FieldFox recently and I took some drift measurements, although those were on a short rather than a load. The 8510C blew the FieldFox out of the water, lol. Given the TDR, this might be because the standard itself was temperature sensitive and the FieldFox ports are piping hot, but still.
https://jjoonathan.github.io/plot_drift.html
In case they are useful, here are a bunch of different standards measured by the two instruments.
https://github.com/jjoonathan/cal-std-8510-fieldfox/blob/mai...
EDIT: Also worth mentioning, I recently upstreamed my nice 8510C driver into scikit-rf!
https://scikit-rf.readthedocs.io/en/latest/api/vi/generated/...
Thanks also for the scikit-rf driver - I've an 8753ES as well, and I've used scikit-rf with that, but not yet with the 8510.
If you can get your hands on an 8360 source, it will synthesize both the start and end of RAMP sweeps (starting on the second sweep -- it looks to see how far off it was and corrects it), and you will also have significantly faster STEP sweeps.
EDIT: obligatory, pictures https://photos.app.goo.gl/KwvnMGm5GyvnmgE58