The video game "Super Hexagon"[1] is a surprisingly interesting demonstration of this limitation, and how the mind tries to workaround the problem. The higher difficulties ("hexagonest"[2]) seem impossible at first. In the easier levels you can react, but now the time it takes to see the scree, parse the simple graphics, recognize that you need to turn left to avo##CRASH##...##GAME OVER##, You quickly learn that if you think about the game, you are guaranteed to fail due to insufficient reaction time. Winning requires a mental state possible related to meditation: don't think, let the muscle memory do it's job (maybe from a faster path limited to the spinal-chord and brain-stem?).
[1] https://www.youtube.com/watch?v=UrPXBSh4mqU
[2] I finally got a winning 62.09s a few days ago, after many months starting at a high score of 58.57
https://youtu.be/TWiMnENdcyY?t=56
At this speed the player is reacting (and hitting with correct timing) about ten notes (arrows) per second. This is possible by memorizing and reacting to group patterns instead of individual notes, but more importantly, reacting unconsciously.
A player can play at this level without conscious attention, even while holding a conversation, or while focusing at different parts of the screen, or spaced out. It feels like the fingers play by themselves.
This is so automatic that the player may have little feedback on how well they are doing. They may think they are making blunders and about to lose, while "the fingers" have correctly hit every note so far.
Definitely feels like a faster neural path was built.
Super Hexagon doesn't allow any of that. The game is randomly generated each time, making memorization impossible. It's 100% reaction against the incoming wall pattern, with no pauses. Except you don't actually have time to "react".
I recommend this[2] essay that does a much better (and more poetic) job of explaining how uniquely Supper Hexagon interacts with human perception.
[1] https://www.youtube.com/watch?v=JRUwpUqSFbI
[2] https://problemmachine.wordpress.com/2015/04/11/hexagon/
First of all, players don't need to memorize the charts. Take a good player, give him an entirely new chart and he will perform much better than any beginner ever will, even with memorization. Shuffle mode, where the notes are mixed and therefore can't be memorized is just a minor handicap. Much, much easier than memorizing a whole song.
There are two things at work here.
The reaction time is actually not that short. For high level play, players typically set notes to appear about 500ms before action. At that rate, anyone can hit a single note, the difficulty is that there are lots of them.
The difference between experienced players and beginners is that experiences players recognize blocks instead of single notes, just like you read entire words rather than individual letters.
Being able to play a prima vista is a rather specialized skill even among musicians; I've only known a handful of people who can do it really well. I don't think it has a ton of utility (at least I never found it to have a lot) and so it's not something that many people actively develop or train, though.
Rock Band 4 has a "Brutal" mode where notes are almost completely hidden save for a slice of a millisecond at the top, but although it does give a hint of an information that may be processed in the end it mostly doesn't matter, since at that point you can effectively do it blindfolded, which is as a process no different to learning to play the actual song.
This makes Super Hexagon more a game of quick pattern recognition than reaction time.
When I play it I am generally focused at the edges of the screen to quickly identify the next pattern and only using my peripheral vision to maneuver around the walls in the center.
But new versions come out, with new sets of songs, and people play songs of the same approximate difficulty as well. If it was simply memorization, they wouldn't be able to play same or near difficulty songs. They'd have to "ramp up" as they memorized the new song.
Obviously, people's brains are reacting to similar patterns (each song has a "style" and many songs are of the same "style" with things like jumps, diagonals, triplets and so on), as well as other subconscious neural reactions.
When you're playing 9-step (out of 10) step songs, you're not watching individual arrows at all. It'd be impossible. You unfocus your eyes and play through your peripheral vision. Your body reacts. You don't need to think at all.
(So nobody is arguing that the Hexagon game isn't "more random" . They're arguing that it's more than just randomness at play.)
It's really no different that carrying on a conversation while playing a game of casual pong with a friend. Everyone just calls it "muscle memory." The pattern is never the same with a ball bouncing off a table. It's not rocket science.
I think part of the reason is from my experience as an organist: some instruments have a noticeable delay from the time that the key is pressed to when the sound reaches my ears, whether due to delays built into the control mechanism itself, as well as sound delay (the console is sometimes located on the opposite side of the hall from the pipes). I think this has caused my brain to be wired to accommodate these sorts of situations.
Chivalry, for example.
Part of the fun is learning to control the wind up correctly so that you are in the exact place (you being a moving, rotating person AND then being a moving rotating person) somewhere in the future as opposed to "close in until within range/in-scope, then immediately triggering the left-mouse button." You have to continue guiding the attack as the attack is winding up.
Another interesting one is long range sniping in simulator and quasi-simulator FPS games like say, PUBG. There's bullet drop, and people can move toward, away, and side-ways. You've got to place the shot ahead of the target, for a given targets speed (walking vs running, vs in a vehicle), and lead the appropriate amount. The amount also changes with distance (obviously) as well as different guns have different bullet velocities (less velocity = more lead), as well as different scopes (2x/4x/8x/15x) which offer drastically different required mouse movements to get the appropriate crosshair movement (even if the "lead" corrected for magnification is the same).
God, I love our brains. I would love to read a textbook on EXACTLY that kind of learning. Subconscious, neural learning.
Oh, and that's one great thing about PUBG. The combination of low-level muscle memory for maximum speed target acquisition and firing, with the constant high-level mental strain of planning 2, 3, and 20 steps ahead. Solving for "this situation right now" (shot this guy on my screen AND how do I outflank this guy walking nearby but out of sight that is also trying to kill me), solving for "what happens next" (if I fire, will it attract another, full health, group of people that will kill me?) and solving for "how do I keep getting closer to the blue zone to reach the end game without dying."
All of those circuits are absolutely required to "win" in a game with 100 players and only one winner. For the fun of it, I really should write down my other mental processes that go on. Getting further in the game (closer to 100) is all about learning to train your brain into juggling many problems at the same time as well as specifically targeting areas where you are lacking. Like if you're not good at close combat, you can get near the end by sneaking but you will eventually die. And combat at short range, medium range, and long-range combat all have different strategies and tradeoffs. Combat with groups is different that combat with 1 enemy, or a pair.
I can comfortably send morse code at ~25wpm, approx. 125 characters/minute. ('Comfortably' meaning others can comfortably figure out what I am trying to send)
That equals ~375 hand movements/minute, or 6.25/second.
(Granted, I cheat - I use a single lever paddle and a microcontroller to get the timing exactly right.)
I've heard people going twice as fast, though that's very rare.
However, people more skilled than I can use a straight key at up to ~35wpm or so - that's ~9 taps a second, while also getting the timing just right without the aid of a microcontroller. That, in my book, is very impressive.
The first season was played very conciously and slow. After each season, I recognized that certain patterns just came out perfectly without difficulty. Sometimes I was even surprised by myself.
However, what I wanted to say is, it was enjoyable just letting the brain do its job of processing patterns and reacting to them. It was relaxing.
Part of practice is learning how to use muscle memory to handle the fast stuff – faster than you can process at full speed – freeing up your mind to think about the broader strokes. It's interesting to think that activities such as playing music, drawing, painting or practising sports are really a combination or interaction of fast muscle memory, slower rational/strategic thinking, and guiding emotion/intuition.
Used to play (still could if I got back to it) at that difficulty with my keyboard. Got to the point where I could have conversations while playing. I also play guitar so I stopped playing Stepmania to avoid extra strain.
You've just described how people play music in general :P the patterns trigger memory and other internal processes (whether it's recalling verbatim or dynamically based on a new pattern from some combination of scales), this escapes the limitation of input output response time because it's all internal with some arbitrary output delay which we account for by "leading" whatever the physical action is (for an exaggeration think about percussion instruments).
If you are at all musical it's fun to do a bit of introspection here... with anything slightly complicated where you do not have time to think about the position of each note (a pretty unnatural way to perform) you will realise that your fingers (or toes in some strange cases), react to a sort of stream of signals internally... it's probably not even a stream but more of a parallel matching against the pattern from which you pick a stream of notes internally because this is right brain stuff.
Disclaimer: i'm not a neuroscientists or musicologists, I just like thinking about the brain and am slightly musical.
One game that really demonstrated this to me was Crypt of the Necrodancer. It's a rougelike where each turn is the beat of a song, and the tempo of the song for each floor is slightly faster than the previous floor, starting at 120 bpm (1/2 second per turn) on floor 1 and topping out around 180 bpm (1/3 second per turn). For the slow songs, you have enough time between beats to fully plan your next action, but at some point, as the tempo gradually increases from floor to floor, you hit a point that this is no longer possible, and your brain has to "switch modes" to planning several beats ahead and thinking in patterns instead of individual moves.
From what I recall, to finish the final level I was relying for the most part on 'muscle memory' in relation to the familiar patterns. I would visually interpret what kind of pattern it was, my position relative to the pattern and then there seemed to be a conscious disconnect to the actions I performed to get through the pattern.
I think it's interesting because the actions I'm referring to weren't merely a sequence of button presses performed as fast as you could. There was a critical factor of very small differences in durations between actions and how long you execute actions for that seem (and I assume are) completely beyond my conscious abilities.
As for the game you played, it must have been the DDR series. You can't "finish" Stepmania since it's an open source rhythm game where the community has created thousands and thousands of songs to to play.
I've spent quite sometime on super hexagon myself.
If you want to blind A/B test yourself, run this:
DELAY='0.1'; VALUES=(0 0); VALUES[$((RANDOM % 2))]=$DELAY; alias test_a="sleep ${VALUES[0]}"; alias test_b="sleep ${VALUES[1]}"
Adjust DELAY as you want, and then use test_a and test_b and see if you can guess which is which, then run alias to see for sure.https://jsfiddle.net/zs8bncxj/1/
I can still get it right pretty much 100% of the time, but the difference does feel a lot more subtle than what I was seeing in iTerm.
(If you change the delay, you need to hit randomize to make it take.)
Edit: Also we should keep in mind that we're not testing the latency we set in either test. We're testing that delay plus the delay from I/O. So 100ms might be indistinguishable. But by the time we pile everything on, we're getting closer to 200.
The result is the same as far as UI design is concerned; don't assume that you can get away with 100ms; most of that leeway has already been wasted by the OS and hardware.
Personally I can distinguish every time at 0.06.
I find it difficult to believe the rest of the hardware loop has 40ms latency, it would be difficult for smooth rendering to occur if it did.
I suspect that part of this is 'training' yourself but also what the researchers meant. They may very well have meant that above 100ms is jarring and noticed as a delay but below 100ms is not seen as a delay rather than truly being unable to notice in an A/B test.
I suspect this test would be more difficult if it sometimes did A/A and B/B.
Why would I/O latency have any effect on the smoothness of rendering? Are you sure you aren't thinking of throughput?
Remember: TFA just told us that many keyboards on their own add more than 40ms of latency. I definitely wouldn't have guessed that, so I'm very reluctant to entertain any certainty about the rest of the system having low latency.
I can get down to 24ms but no less... the weird thing is that 24ms is still completely obvious to me (clearly shorter but obvious in comparison to no delay), with a single test I can see which variable has the delay every time, but 1ms less and I can't... which makes me suspect it's being quantised due to any of the various things in between sleep and the output, display, driver, X, terminal emulator, CPU etc.
With that it's actually possible that my 24ms is larger than 24ms and is also being quantised to a larger duration (but not larger than 100 for sure).
It would be interesting to be able to test with some dedicated hardware.
It's not quite a keyboard delay test but more of a general human reaction time test.
The best I can do is 160ms after 5 tries on a 2560x1440 60hz IPS panel.
I'm using Alacritty, which is supposed to be really fast and everything!
Try it out.
Look at this video on Youtube from Microsoft Applied Sciences Group: High Performance Touch.
https://www.youtube.com/watch?v=vOvQCPLkPt4
You can plainly see that 100ms is an eternity. While keyboards aren't the same thing, but high latency is noticeable. I had to change a keyboard because of the near eternity of when I pressed a key and when it registered on the screen. The difference between the old and the new is quite stark.
Unfortunately, programmers/UX designers have a tendency to generalize this rule and use it to excuse slow user interfaces.
EDIT: Which the original author said better than me:
> Another problem with this line of reasoning is that the full pipeline from keypress to screen update is quite long and if you say that it’s always fine to add 10ms here and 10ms there, you end up with a much larger amount of bloat through the entire pipeline, which is how we got where we are today, where can buy a system with the CPU that gives you the fastest single-threaded performance money can buy and get 6x the latency of a machine from the 70s.
That's how it goes with my ears when recording as well. Most "Live feedback" mechanisms for guitar programs (heck even computer-made guitar hardware) have about 50ms of latency, which is quite disconcerting when you're doing something heavily timing-based. 10ms is almost imperceptible to me (sounds more like the tiniest slightest reverb delay) and 5ms might as well be realtime as far as I'm concerned.
An example, in Windows 10, I could go to the built-in mixer, turn on the "Listen to this device" and play with my guitar plugged right into the computer, and you're looking at about 100 ms or so of latency. Completely unacceptable when you're listening to a metronome to keep on time. Alternatively, I can go to the Realtek HD Audio Manager in the taskbar (because I installed the actual RealTek driver package instead of relying upon the Windows drivers) and un-mute the line in, and even with an amp and mixer board in-line before the computer, the feedback is essentially instantaneous, which allows for pre-baked instant reaction times (eg muscle memory) to work and record in time with the metronome (or backing track.)
Another fun test you can do, if you have an analog mixer, is to run some input signal into the mixer, then to the PC, and then from the PC back out, and monitor both the original input and the signal from the PC at the same time. OS-supplied drivers typically make it sound like you're in the Grand Canyon.
But even with very expensive low-latency gear, you can still tell if you put one signal into the left headphone channel and another into the right. If there's actually zero latency, you'll perceive the sounds as coming from directly in front of (some people perceive as directly behind) you. It'll sound just like mono. Any latency, given the same volume level, will be perceptible as a difference in "location" of the sound.
My unscientific experiments suggest that you can perceive down to about 1ms latency quite clearly this way across most of the audible range. The minimum is frequency dependent (as you'd expect) with lower frequencies being less sensitive.
Presumably the human brain has evolved to be extremely good at arrival time comparisons (maybe even doing some sort of phase-difference stuff) as a way of triangulating the origin point of sounds. There are doubtless other specialized ways in which the brain is equally sensitive, so the 100ms rule of thumb seems pretty naive to me. 100ms might hold true for the "main loop" of the human brain at the level of conscious thought, perhaps.
What reasonably-priced DAC would you recommend?
In the market for one currently.
> Not to mention that 100 ms is musically a 16th note at 150 bpm. Being off by a 16th note even at that speed is – especially for percussive instruments – obvious.
The auditory system is especially good at processing serial data, and it has a better timing resolution than the visual one.
People with visual-auditory synesthesia are much better at tracking patterns in flashes of light for example (because the hear the flashes).
Of course there is some threshold of latency when it starts to become noticeable/disconcerting for some fraction of people and probably quite a lot more when half of all typists start to notice. Anyone can definitely feel this on terminals connected to very remote machines.
This is an interesting topic and am glad that Mr Luu is looking into it. The measurements described are absolutely a valuable first step towards understanding what latency really means for human factors in typing (first, what is the f-ing latency). I expect the folks at Logitech and other places have significant proprietary research on this stuff, but maybe not?
I can see how high latency is acceptable if you're only putting in new characters, but as soon as I have to fix a typo or anything more complex, I get really mad really fast if the latency is anything beyond (ballpark estimate) 250 ms.
[1]: http://www.press.uchicago.edu/Misc/Chicago/093185.htmlhttps://forums.blurbusters.com/viewtopic.php?f=10&t=1134&sid...
Some people in that thread are tolerant to 10ms...
If it was audio cue, I'd point you to "Love Live School Idol Festival", a rhythm game which "perfect" judgement window is some 32ms. Decent players will get perfect most of the time.
There are plenty of examples on YouTube if anyone's interested.
Similarly, limited frame rate makes it impossible to keep the image at exactly the same place on the retina during smooth pursuit eye tracking, causing perceived blur that wouldn't be seen with a real life moving object.
Seeing a difference with 1000fps video doesn't mean we have 1ms temporal resolution perception, it just means some test signals allow conversion of temporal information to spatial information by eye movement. It doesn't generalize to arbitrary visual events. To investigate keyboard latency sensitivity properly there's no substitute for ABX testing of actual keyboards.
There's a difference in constant, but consistent lag (delay) and variation of lag variance(lag)>0. If one note is a 16th note off that is lag variance.
Not to mention that 100 ms is musically a 16th note at 150 bpm. Being off by a 16th note even at that speed is – especially for percussive instruments – obvious.
On the other hand, if you told me to strike a key less than 100 ms after some visual stimuli, I'm sure I couldn't do it – that's what "reaction time" is.