I think the rule of thumb I used to go by when recording myself playing guitar was that 20ms was a noticeable delay, 100ms was sort of tolerable and anything more than that was enough to make my picking lose synchronization.
I remember hearing 16 ms being the general threshold for most people recognizing out of synch audio.
I feel like that would be enough to be basically perfect, though. If only we could cut down on everything else in the chain adding its own latency.