If you look at professional distributed audio systems (Dante, AES67 etc) you'll find that they all require PTP support on the hardware to achieve the required timing accuracy, so yes, you need <1ms to get to the point of being considered suitable if you are doing anything which involves, say, mixing multiple streams together and avoiding phasing type effects.

However, it very much depends on what your expectations are, and how critical your listening is. If no one is looking for problems, it can be made to work well enough.