There is a TED talk (I don't remember the presenter's name, talk date, or anything else) where the presenter basicaly asks:

"When your children grow up and asks you why you didn't follow your dreams, do you want to answer - It was because of you?"

At the same time, yes, it's very hard.