1. It's collective punishment. Unethical unless absolutely necessary. 2. It would be ineffective. In most cases in the west the relatives of people who ended up blowing themselves up didn't know about their involvement in terrorist organizations and didn't condone their actions.
And it would encourage people to be more alert about suspicious terrorist activity within their family.
It's also better than a poke in the eye with a sharp stick. So what? We don't have to have any of those things.
Also, collective punishment is not better than massive inconvenience. Convenience is never an excuse for doing something immoral.
And close relatives would have a greater incentive to report him/her if they suspect something.
Of course, you could consider a different sort of penalty such as fines that would bankrupt his/her close relatives.