For me that falls under taking responsibility for my own life. If I were to be constantly tempted it would be a signal something is wrong. I'd discuss this with my partner. Perhaps we should break up. The alternative is staying with him/her out of pity. Pity is not love, staying with someone out of pity is worse than leaving them imo. If my partner was constantly tempted but staying with me out of pity it would be the ultimate insult. I want my partner to be with me out of pure self-interest, because I'm the best choice, her favorite human being, her ultimate partner, the best she can get. I don't want to be a compromise.
I think that there is even a case to be made that leaving while you have children is better for them than keeping them in an unhappy marriage. You are certainly not giving a good show of what love is all about.
I do agree about the feeding, you certainly have the power to feed and deprive certain feelings.
A certain statement like "You certainly are not responsible for other people.", is misleading and unhelpful because it suggests people who are tempted by relationships outside the home should go ahead and pursue them, even if it means feeling bad for being irresponsible towards your children and family.
Your point is you are being responsible because it makes you feel good, taking these feelings for granted. Yes, nature gives them to us to begin with, but they must also be nurtured. Between the good dog and the evil wolf in all our hearts, the one that wins is the one we feed.