It indeed can't be too big because the shooter takes the same or greater momentum in the form of recoil. Conservation of momentum applies to the process of accelerating the bullet.