For this to work you unsolve the key-sharing problem. Everybody who will ever want to communicate with anybody must agree in advance shared secrets with every such party and then retain them indefinitely. You assert the resulting scaling "is harder, but it's not impossible" and I'm sure that by some technical definition of "impossible" this statement is even true, but in practical terms it's never going get done which is why such systems are used (if at all) only by secret agents.

If you decide well that's OK, we can do something simpler - everybody will have an infinite set of aliases like "Zanizabo" or "Fylnatis" but they won't be one-to-one shared secrets, alas Mallory simply hears you say "Zanizabo" and she says "Hey, Zanizabo, call Oxymoron" (Oxymoron is an alias Mallory has chosen for this purpose) and soon discovers that Bob answers so now she knows who you wanted to talk to.