I was politely asked to leave my local group under similar conditions (non-belief causing friction), though I was only a webelo at the time.

I would probably feel out the group first if you intend to do this, it wasn't a pleasant experience at the time.