In this case, the real alternative to the task switcher is flicking through multiple pages of apps, which surely is more difficult than a simple row of recently used apps. It's hard to imagine a pinching gesture that would be usable here, without interfering with the current app.
However, if a user is switching between three or four apps for a given period of time, those apps may very well lie on very different pages of the Springboard. The most frequently used apps are not always the apps that matter now.
Even when you would need to swipe in the task switcher, surely it is easier to find an app among 4 instead of 16?
(Maybe the task switcher really is only useful for a small subset of use cases, in which case your position would seem to be the logical one.)
The task switcher lets you switch back and forth between apps one handed using your thumb. I think that's part of the rationale for it. Also, the Least Recently Used algorithm will dynamically make a short list of the actually used apps in a user's collection. Most users probably spend 80% of time in a small minority of their apps.