I'd be ok with an app by a third party which does not automatically gets network access on my phone explicitly or implicitly, so I know my data doesn't leave my phone.
AFAIK, except when you enrol in a study, none of that depends on network access.
[1] https://habr.com/en/post/579714/
EDIT: Found the link to the previous HN discussion about this: https://news.ycombinator.com/item?id=28637276
Recall that Google got to claim their street view cars wardriving was unintentional, and just paid $13M to make it go away, after using the data for about a decade [1]
https://www.cnn.com/2019/07/22/tech/google-street-view-priva...
The trust you have in the system is not the trust I have.
Not at all. What you are saying is that you don't want to lock the front-door because nobody has illegally entered your house yet. In contrast, I want to lock my front-door for obvious reasons. No paranoia. It's just a different mentality of dealing with security.
As long as no health data leaves my Apple Watch + iPhone, I am OK with local deep learning (or other models) running on my devices.
I do wish there were International standards and local laws supporting: all data staying on users' devices unless explicitly shared with health care providers, and also agreed upon standards for anonymizing explicitly shared data for the public good (research, AI training data, etc.)