Sure. The first time around at megacorp I left shortly before a reorg that demoted my manager to IC status. He was still a manager when I returned but a manager with no direct reports.