You can even get creative and say "Yes" in such a way that demonstrates you're not serious about the commitment. I used to do landscaping work in college, and when my boss discovered I was good with numbers, I took over part of the pricing for new jobs. When someone asked for a quote for something we could do but didn't want to (e.g. building a large retaining wall, mowing a 50 acre field in an inconvenient location), I'd give a "go away" price (usually by tripling our labor rate and padding the estimate) that would signal we didn't really want to take the job but didn't hurt our reputation in the same way a blanket "we don't do that" would.
This let the owner call back and say "Yes, we'd love to mow your field. We've run some numbers, and based on the location and the size of the job, it's going to cost you $2500. Oh, you'll pass? So sorry to hear that, have a nice day." Most of the time, people passed. The couple of times that someone took our "go away" price were rare enough that we were able to keep our focus on residential landscaping (which has much lower overhead) but frequent enough that we could still advertise we did larger scale jobs.
A qualified "Yes" will leave doors open when a "No" would slam them shut, and it's good to know how to say one in any field.