The slave's own jitter tendencies might be important - but then again, where do they end up? Mostly in the analog /dev/null sinkhole - the sampling frequency is far above the audio spectrum, the jitter components are too, mostly (unless it's low-freq drift), and the output of your DAC has a steep downpass filter. Jitter components are obliterated.
That's assuming that jitter would matter at all even if it was not filtered out. But if a DAC is flickering that badly that you could hear it, it's probably a piece of trash anyway.
Make sure the actual digital-to-analog conversion (the last link in the digital chain) is done well, and don't worry about anything else. Yes, you can screw up DAC design and execution, but nowadays most DACs are very good. This is not your grandpa's radio.
"Eric Benjamin and Benjamin Gannon, "Theoretical and Audible Effects of Jitter on Digital Audio Quality", Preprint 4826 of the 105th AES Convention, San Francisco, September 1998"
or
"The Effects of Sampling Clock Jitter on Nyquist Sampling Analog-to-Digital Converters, and on Oversampling Delta-Sigma ADCs, 87th Convention of the Audio Engineering Society, October, 1989"
or
Ashihara and Kiryu, etc (2005)
"About 25% of the listeners detected jitter when its size was 500ns. When it was 250ns, however, no listener could discriminate the sounds."
...
"Nishimura and Koizumi made attempts to measure actual jitter of various DA systems during reproduction of music signals. They could not detect any jitter larger than 3ns in their measurements."
...
"So far, actual jitter in consumer products seems to be too small to be detected at least for reproduction of music signals"
...
CONCLUSION "The threshold values seem to be sufficiently larger than the jitter actually observed in various consumer products."
The only way to make jitter audible is to do it deliberately, to the extent that the DAC would have difficulty even locking to it. At that point, it would likely end up dropping samples, and we're not really talking about digital audio any more.
Audio marketing has been using crap like "digital jitter" as one of their snake oil sales tools for as long as digital audio has been commonplace. Don't give it even the slightest amount of publicity.