Android 7 (lineage/cyanogen at least) can do this.
Settings>Sound>Do Not Disturb>Priority only allows
You can limit notifications to starred contacts only for calls and also SMS, and then only star the contacts you want to allow.
In Android Do Not Disturb mode can do this. You can set it to "priority only" and define who is the priority contacts. iOS should have a similar function.
Poor man's version of this: set your ringtone to a silent MP3, then apply custom tones to those people you actually want to hear.
Could you describe briefly how to configure that state of affairs? What OS do you use?