Another fun test you can do, if you have an analog mixer, is to run some input signal into the mixer, then to the PC, and then from the PC back out, and monitor both the original input and the signal from the PC at the same time. OS-supplied drivers typically make it sound like you're in the Grand Canyon.
But even with very expensive low-latency gear, you can still tell if you put one signal into the left headphone channel and another into the right. If there's actually zero latency, you'll perceive the sounds as coming from directly in front of (some people perceive as directly behind) you. It'll sound just like mono. Any latency, given the same volume level, will be perceptible as a difference in "location" of the sound.
My unscientific experiments suggest that you can perceive down to about 1ms latency quite clearly this way across most of the audible range. The minimum is frequency dependent (as you'd expect) with lower frequencies being less sensitive.
Presumably the human brain has evolved to be extremely good at arrival time comparisons (maybe even doing some sort of phase-difference stuff) as a way of triangulating the origin point of sounds. There are doubtless other specialized ways in which the brain is equally sensitive, so the 100ms rule of thumb seems pretty naive to me. 100ms might hold true for the "main loop" of the human brain at the level of conscious thought, perhaps.
That's how it goes with my ears when recording as well. Most "Live feedback" mechanisms for guitar programs (heck even computer-made guitar hardware) have about 50ms of latency, which is quite disconcerting when you're doing something heavily timing-based. 10ms is almost imperceptible to me (sounds more like the tiniest slightest reverb delay) and 5ms might as well be realtime as far as I'm concerned.
An example, in Windows 10, I could go to the built-in mixer, turn on the "Listen to this device" and play with my guitar plugged right into the computer, and you're looking at about 100 ms or so of latency. Completely unacceptable when you're listening to a metronome to keep on time. Alternatively, I can go to the Realtek HD Audio Manager in the taskbar (because I installed the actual RealTek driver package instead of relying upon the Windows drivers) and un-mute the line in, and even with an amp and mixer board in-line before the computer, the feedback is essentially instantaneous, which allows for pre-baked instant reaction times (eg muscle memory) to work and record in time with the metronome (or backing track.)