Management has the power to say that something isn't good enough and make it a priority. They also have the power to hire employees or consultants if the current team isn't capable of doing it.

Project management _definitely_ has the power to dedicate time to fixing performance issues.

Metrics can be gamed, but certain metrics - such as time to interactive, and time to fully loaded - are fairly well in line with what users actually care about. Even if they're gamed, a project manager can say, "This still feels slow to use. Dedicate the next (sprint|cycle|month|whatever) to performance work."