Doesn't matter != matters least.
The performance of the desktop on the PS3 and PS4 is less important to the gamer than the performance of the games, also known as the primary purpose of the device.
However, that doesn't mean that it is unimportant.
Interface performance is really fucking important, and I sure hope people like Sony and Microsoft care when it comes to their consoles.
People like EA and the notoriously terrible Battlefield* interfaces obviously do not care. (The original Battlefield 1942 would switch from your native resolution and refresh rate and color depth and put the screen in 800x600 16bit 60hz every single time you hit the escape button.)