We're prepared to do this if necessary, but most of these users aren't actually using the app. So the only apparent damage is them getting upset that the app keeps reappearing on their phone.

If that’s what’s happening and it’s not happening through the play store then I’m afraid there may be nothing you can do about it.