I think there should be a compromise. Use your own config but be ready to switch quickly to a more "common" config and know how to use it (whatever common means in your context, just adapt to your team).

For example I use i3, and I switch ctrl and caps lock. During my first pair programming experience my colleague was struggling with all that and I was feeling quite guilty ("maybe this need to fine-tune everything is just immature after all"). After this I made sure to be able to switch quickly to a "vanilla" Gnome setup.

Although coming back to Gnome after using i3 is quite easy since the Gnome paradigm is quite universal, there are some alternative "configs" that I am afraid to adopt, such as exotic keyboard layouts ("bepo" in french instead of "azerty"). Losing the ability to type on this de facto standard layout would be quite problematic.