This is a tough one. If asked, most everyone will complain about delays. Just like, if asked, most people will complain about the color choice. Doesn't mean it is the best place to focus on at first.

In areas where the latency has been solved, don't make the options worse for the user. There are bare minimums, but they are usually hard to mess up. Such that, if you are bringing something new to the table, focus on that first.