Perhaps this has changed, but my experience was that the installation wouldn't proceed without granting access. I could prevent the app gaining access to my contacts by using the Android home/work profile so that it would only see an empty contact database. There are other tricks you can use on Android, but I never got it to work on iOS.