That's only true because mobile started from scratch a few years ago. You just see innovation because this market (for "smartphones and tablets") barely existed 6 years ago. Obviously this will tone down very fast as performances stop increasing significantly. You see that the latest ARM chips are only marginally more powerful than the previous ones in terms of operations/Watts.
So, they will, also, get only "a little faster every couple of years." Do not kid yourself.
Do you actually visit your local PC stores regularly?
Because while it was not unusual to see full sized towers on display, these days it's rare to even see a standard ATX case. The typical form-factor even for the desktops have slimmed down massively, and a large number of the devices are now "all in one" models with the PC and monitor in one.
That's actually why I agree with you about mobile - the performance _most_ people needs is low enough that the form factor has been steadily going towards mobile, and yet sales are still anaemic while mobile device sales have sky-rocketed. With the rise of Android "TV sticks", coupled with increasing support for mirroring/streaming display from your mobile devices, and/or HDMI/MHL ports and bluetooth keyboards, the desktop is going to be redundant for most people within a few years.
The irony is the desktop trend towards small devices, bluetooth keyboards and "all in one" models are accelerating the demise of the desktop more than it is propping it up, by making people used to wireless keyboards and not having a big box PC.
Multicore is also bigger in PC's than smaller devices, but that's mostly trickle-down from server-space.
Innovation will be driven by mobile (both phones and tablets). It requires a lot more innovation to build these smaller devices that operate all day on a battery.