I'd say based on the number of friends I have who use old or cheap Android phones with terrible latency, it really was the touch screen and UI/UX that played the main role.

Sure, for many, iPhones are still preferred because of the low latency, so it matters, but I suspect the iPhone would've done just as well if the latency was bad. The competition was about finger vs keys/pencil, not latency.