I have seen exactly that happen multiple times. The in office workers were core team, the from home workers were invariably "secondary citizens". In terms of what tasks they got and how much influence they had. They were first to go when the company needed to lay off people. They were forgotten about when splitting tasks, making decisions, starting meetings, dealing with issues.

And second, emotional matters. The productivity does influences how team and management treats you, but so does "connection" and crap like that. If you have jerk tendencies or lazy tendencies, people tend to excuse those much more if they know you in person. Bonus if they perceive you as cool or funny.