Switching apps, latency for apps to restore state once you switch to them, latency for the camera to save a photo, latency on pretty much all interactions is noticeably better.

I'm not talking about dropped frames, Apple have always been good at ensuring a fluid user experience, but there are lots of points where apps can take a few seconds to do something on older hardware, or do it nearly instantly on newer hardware.

Does a few seconds make a difference? To my life, no. To my enjoyment of using the device, yeah. Whether it's worth frequent upgrades or not is a value judgement we can only make for ourselves, but there are many perceptible differences devices more than a few years apart.