Thermoelectric elements add resistance. You are literally adding a giant resister between the hot and cold side, which decreases the thermal Flux for a given temperature difference.