It may be up to the merchant, technically, but in reality systems such as Authorize.Net allow you to configure it at the gateway to reject transactions based on various AVS criteria. Many of my clients also only allow shipping to the AVS verified billing address as a way to weed out much of the fraud. For many products, only allowing shipping to the billing address is a reasonable restriction.