This shouldn't be surprising. Consumers of rape fetish porn are unlikely to actually rape someone, but actual rapists are likely to consume rape fetish porn. Furthermore, in countries where porn of all kinds was legalized, including child porn, enjoyed falling rates of child molestation.
The OP is making some of the same old arguments people have trotted out against porn for decades, just couched in pseudo neuroscience. The fact is, we don't understand these relationships to the degree the OP needs to justify his causal claims.
On the other hand, children are extraordinarily vulnerable to many forms of abuse, this among them. In a system of laws one of whose explicit purposes is to afford those vulnerable to mistreatment protection under law, I don't think it is on its face unreasonable to argue that children merit extraordinary protection as well. While there is perhaps a fair question to be asked around whether the sort of law under discussion actually serves that end, I would at the very least suggest anyone raising the question in a serious way be very well prepared to answer objections and counterarguments of every imaginable sort.
While it would be a stretch to postulate that the effect of child pornography on those disposed to it exactly matches that of drugs which provoke massive dopamine release on those who take them, the effects of dopamine on sexual behavior in the human male, and its role in the brain's reward system in general, lend I think considerable plausibility to the hypothesis that engaging in pedophilic behavior centered around child pornography, and obtaining sexual satisfaction thereby, tends to potentiate further engagement in the same behavior.
[1] http://www.sciencedirect.com/science/article/pii/01497634940...
[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2696819/