I agree with you, there is no good reason why a career should suffer from working remotely in a largely non-remote office culture. But a lot of dumb things happen in corporate environments when they shouldn't.