ToS violations may not be "the" reason, but it is enough of a reason to force Google's hand, whether or not they have other reasons.

Exactly how long do you think, say, Viacom woulds tolerate a high-profile, name-brand YouTube app that blocked ads and supported video downloads? And what do you think Viacom would do to Google if Google's response to that app was turning a blind eye?