You can't easily tell a coworker to go away when they start talking to you at your desk. You can mute your notifications and schedule calls. Sounds like you have bad organisation skills.