I think it is unethical to take away the livelihood of someone if the only reason you are doing it is to add even more zeros to your already fat bank account.

Also, you are assuming the only one being rewarded by layoffs are executives. As someone dutifully putting away money for retirement in a 401k, I am the primary beneficiary of such layoffs in their stock prices(assuming the company is publicly traded). Should I feel guilty that my greed has caused layoffs?

Even if one would agree to judge things within your frame, i.e. employee's share vs. employer's, you still have a moral dilemma with allocating resources wisely. Judging from their past income spending practice, employees will most likely consume, whereas employers will most likely invest. I hardly see any reason against supporting the later kind of players.