Even if you designed a pathogen to do that, once released into the wild it would almost certainly spread to other populations.

Viruses and bacteria already can jump zoonotically (between distant species), it would be trivial to adapt to within-species targets.