Setting a server_name is something you need to opt into. So I feel it should be required to match to serve that vhost. If you didn't want it to apply only to the right vhost, don't configure it to require a specific vhost?