I said no to things and world did not ended. You absolutely can say no. If you can't, you need to change the team or job.