Legally speaking they need "reasonable cause" to stop and frisk people on the streets (how that works in practice is a whole different discussion). At the subway entrance they can search all the bags they want for any or no reason.

You can say no, but then you shouldn't walk on the streets.

I smell sarcasm! :) But seriously, riding the subway is not a right. They make rules for the subway and enforce them at their own leisure. But it's 100% legal to turn someone away if they don't comply with rules they make up themselves. MTA is a private entity.