DTMF based challenge is probably easier. Even requiring callers to dial 1 to ring through is enough to keep robocall campaigns from ringing your physical phone.

I've been thinking about issuing my friends and family priority access PINs. If they save it to their contact entry for me after the number, I can entirely gate access. e.g. "5558675309,9876" Especially if combined with a time condition e.g. 11 p.m. to 6 a.m.