I feel like it's worth it to remember people's names. I also don't think there's any evidence that indicates being better at remembering one thing makes you worse at remembering others. In fact it might even make you better, as you're sharpening your brain, so to speak. (If there's evidence to the contrary please feel free to share.)