This 3% or whatever from 0.5^n is a lower bound. To get close to it, all your other relatives would need to come from completely somewhere else. But in reality, many of your other ancestors were probably related to him, because they had lived in his valley since they invented farming, or whatever. So you probably have some more of his genes from several great, great, great grandmothers who were related to him.