It could be the case that the physical button is trapped directly by the secure enclave system and the goal is not to prevent an app from accidentally fooling you in the UI, but to prevent any iOS software to silently ask the secure enclave to authenticate something without a physical confirmation directly from the user.
A fake pay button couldn't do anything more with the Apple Pay system modal UI than present it. It can't trick you into completing it. The side button click adds nothing. The side button doesn't stop it from presenting a fake payment UI. And furthermore the App Store is a first party app so requiring the side button there can't possibly be to protect from nefarious apps.
A better theory is they just want total UI consistency. Since they need to use the side button for in-store Apple Pay, that wins for the App Store and in-app purchases as well. But even that doesn't quite make sense either, because then why are they requiring it for installing free apps?
As an aside, I don't like having to press the side button for Apple Pay on my iPhone X. I usually have to shift how I'm holding it to get the proper leverage to double-press the button. Double pressing the home button was way easier.