While I do sympathize with the homeowners when it comes to resenting poor behavior by visitors, that thing you are calling "his backyard" is not his. He never paid for it.

English Common Law encodes many centuries of traditions about easement for travel. This topic is not a new thing. That an area might bear ever more visitors over the march of years is just a fact of life.