That's right: once a certain threshhold of compute performance is reached, the excess is only useful for specialists and gamers.
The other element is that mobile devices are optimised for power consumption / battery life and also for heat output. Desktop devices generally have neither of these concerns, unless they strive to be 'green' or fanless products.
That doesn't mean portable devices won't replace them. Laptops began to replace desktops a few years ago, the same way PCs replaced mainframes.