Depends on the application. Your brain can notice the delay when playing something like digital drums after ~10-15 ms. Something with less 'attack' like guitar has a bit more wiggle room, and ambient synths even more so.

edit: also, vocals are most latency-sensitive if you are the one singing and hearing it played back at the same time.

Recording instruments while monitoring them starts becoming really difficult after ~10ms. 2-8ms is ideal in my experience