Would you watch an employee closely if their performance dipped after they started a hobby game development project? Or night classes? How about if they just had a new born?

Do you also acknowledge the weeks their performance is better than average? Do you compensate them for it, or is it just a given?

You have the right to be concerned about someone's time but only while they are at work. Your employees get to do what they want outside of work hours.

I'd wager those X hours a week in the office are often not the most important part of your life, why should it be for them?