When a 48 year old buys sex from a 15 year old then we can treat it exactly in the same manner as if a 48 year old would want buy e.g. inherited real estate from the same 15 year old - we simply acknowledge that the minor isn't able to grant informed consent and is likely to be exploited, so we don't consider the transaction consentual.
A 68 year old buying sex from a 18 year old - why not, it happens e.g. in marriage/"golddigging" scenarios? I mean, I can at the same time morally dislike some behavior and also acknowledge that I do not and should not have any right (moral or legal) to prohibit that behavior if they wish to do so.
So, there are some people who'll say that any paid for sex is wrong, even if all people involved are freely able to make the choice. Let's ignore those people for a moment.
> Why should it matter with whom one has sex with if all parties consent?
48 year old buys sex from a 15 year old? Neither are in debt, or have drug addiction. Each say it's their free choice. Is that a problem?
Person buys sex from other person. The person selling sex has a drug addiction, and is using sex work as a method to buy drugs. Is the seller able to give consent in this situation?
There are many more, and these feel like moral judgements to be made, and it doesn't feel unreasonable to try to work out where the border between acceptable and unacceptable is.