The obvious downside is that you're missing out on more diverse perspectives. People who spend time outside of work on things unrelated to their field, whether that's as a caregiver or unrelated hobbies, will look at the world differently than someone who dedicates most of their time to their craft.