They are fearful that people with your stance will give companies a better argument for less remote-work. It's one thing when executives talk about it, because they're "not on the ground" and "just want control" or "don't understand individual contributors", but when peers like you support non-100% remote, it's a real worrisome prospect for them.