Why would that be immoral if you have negligible negative impact? It would only be a problem if everybody else will begin doing it. Which does not happen. Having children, hunting in the woods, owning a big house. Eating meat. Not being a universal example for every person is not immoral. People have different needs, and one person has more impact on the environment than other (but usually still negligible). If you want to limit yourself in this particular way, fine, but this isn't particularly moral thing to do. It is more of a feel-good thing to do.
I suppose my desire is partially to exit a commercial food system that I know is unsustainable. That system doesn't look like it will become sustainable any time soon, and I don't want to vote for it with my dollars and contribute to the issue with each meal.
I also don't necessarily do it to feel good. It feels bad at times, otherwise neutral. It feels like doing too little. There's no warm fuzzy feeling from half measures like this. I wish I could do more, but I struggle to figure out how.
Having kids robbed me of the selfishness I required to isolate myself from the externalities of my lifestyle. To be honest, some days I wish I still didn't worry about it.