I think a big part of it is simply loading in resources and the like. Smartphones try to keep a _lot_ of apps in memory, so there's a lot of switching around.

Another thing is that Windows would put certain native widgets (like file selection) on a higher priority than other program code. Android, at least, tries to put as much stuff in userspace as possible, so you might be experiencing the reality if everything run at the same priority.