There are two other ways scrolling on a 60Hz vs 120Hz phone would allow you to tell the difference: the increased smoothness of the scroll (not related to latency at all) and how far the content lags behind your finger. The content lagging behind where you initially touched does allow you to experience the latency, but that is more of a visual only experience than a temporal one.
Apple reduced the latency in its pencil from 20ms to 9ms. It's well regarded that the reduction of 11ms in latency is very much noticeable.
It's different for where you don't have same senses for both. "Pressing" a key is feeling + sound but reaction is change in image, you don't see keyboard and screen at same time so it feels more tolerable.
It's not really more tolerable, unless you never experienced low latency here, and your brain is used to the effect of a key press being perceptually immediate.
Case in point...
My previous iPhone had 60Hz refresh rate, that equates to ~16ms.
My new iPhone has 120Hz refresh rate, that equates to just ~8ms.
I can absolutely tell the difference in the display between these two devices when I scroll.