It's part that, but it's amplified by the fact that even in the tiny desktop GNU/Linux usage, there is no one GNU/Linux: a virus targetting a KDE, or GNOME, or XFCE, or—you get the point—something else, is targetting an even smaller part of that already small number of users.

Basically, the level of diversity in the ecosystem, while part of a reason for less widespread acceptance, also discourages any mainstream attacks.