Most of your data (ex. search history) is not reliably stored on your phone, and even if it was it would be much better to train on the server side instead of making your phone do all of the work. Also, they probably train using data from everyone, not just one user.

OK, but apple can store your data encrypted on their servers or on your phone and still train data models... Privacy can be preserved.