I do think your logic holds for many combinations of users and infrequently-used applications. If your "brain loop" works like: press a button, then wait to see the result and completely mentally parse the screen before pressing anything else, then yes, there will be a few dozen frames after any action in which the program can safely run a garbage collector. This is the way we all operate when we first start using a new program. But then, for many users, once they've become familiar enough with that program, they no longer need to think about most of the screens, and can rely entirely on muscle memory to navigate.[1] At this point, any additional latency is certainly noticed. The best example would be typing; at least on HN, the vast majority of us can touch type, and any pauses between typing and characters appearing on screen (such as the incessant pauses I experience with Firefox and iOS Safari) are incredibly annoying. And since typing is an important part of almost any application, that basically necessitates a never-ending "continuous interaction," I don't think you can discount the importance of a highly interactive feedback loop for everyday programs.

I don't really see how you were discussing incremental GC in your original post, but yes, incremental GCs designed to scan their heaps in small bursts do exist and are common in embedded languages.

I wish that modern OSes and programming paradigms had better support for signals/"software interrupts", so that, even if I don't think garbage collecting everything is the right strategy for interactive applications, at least you could say "let me run my GC/other background task up until exactly the moment that the next frame starts," instead of having to manually parcel out work into tiny units while poling a timer.

[1] This is a big part of why I prefer "tacky" 90s Windows and UNIX UIs to modern animation-heavy ones. Once I've "learned" the program, I'm not even consciously looking at the UI, so the superfluous animations just force me to add delays of my own to my muscle memory, instead of letting me work as quickly as my muscles can move. Programmers tend to get this right for keyboard-driven interfaces, but they tend to underestimate our ability to use muscle memory for mouse and touch-driven interfaces.