This is a tough one. If asked, most everyone will complain about delays. Just like, if asked, most people will complain about the color choice. Doesn't mean it is the best place to focus on at first.
In areas where the latency has been solved, don't make the options worse for the user. There are bare minimums, but they are usually hard to mess up. Such that, if you are bringing something new to the table, focus on that first.
Every user group focus test I've ever done on an app with a janky UI has brought it up as a significant "The app is so slow!" issue. On the things I build users hate UI delays.