This would still be the case if you put Windows 10 desktop on a phone. The problem is that even though the Windows desktop has a healthy app ecosystem, these apps work poorly on touch devices. This is true for touch devices with large, high-quality screens (various desktops) and for smaller, more portable, but still high-quality screens (like the Surface). Rewriting an app to work well with touch turns out to be an enormous amount of work for the typical Windows app developer. This is true in the ideal case, for C# devs working with WPF/XAML, but it is much worse for devs working with older apps... WinForms, MVC, etc.
Without the desktop apps, you're left with something that is basically an also-ran in a market saturated by iOS and Android. In the meantime, the desktop experience is still not completely touch-friendly and has suffered from a fair amount of design damage in the process (e.g. two control panels). Any developer that would go through the process of getting their existing Windows app working with a touch UI is going to evaluate porting to iOS or Android, and probably prioritize those options.
[Edit]: I should also add that I really want the opposite of what you ask for. I really want my Android apps to also be available on my desktop PC. It would be really cool if I could plug my phone into a clamshell laptop-like device and have a big-screen experience.