A recovery code snail-mailed/e-mailed to the account holder when they first open the account is the correct way to go, and if they can't provide it they need to go through a lengthy process where many factors are used to authenticate them (verify their physical address, verify their ID, ask to confirm last call records, billing details, etc).
If later it turns out this was a sim swapping attack you can verify if the clerk entered a valid document ID. He can’t do that without having been presented a proper document, so you can tell if he checked.
I wasn't sure how would you solve the problem of verifying the ID card without showing the previously recorded number to the clerk. But simply requiring to every time just punch in the ID (and maybe scan the whole card to check the photo later) could work - if the system only returns a big OK or BAD signal.
Currently here, in Hungary, the clerks just photocopy the IDs though. And there was a big scandal a few years ago (in connection to the ISIL/ISIS attacks in EU) about some groups obtaining hundreds of thousands of SIMs for just a few names.
Your first drawback is substantial, though.
That would be smart criminal with means. I was thinking more of a hood with fat neck passing $20 to clerk assistant to obtain SIM for $5k fraud.
https://arstechnica.com/tech-policy/2019/10/att-employees-he...
So easy for evildoers and so much friction for law-abiding customers.
This is how it works in Poland since September 2019, after some recent SIM-swap attacks. You can swap SIM or get a replacement if stolen only at store showing government ID. It is free of charge with Orange and not always free with T-mobile.
But this has some downsides in real life.
1) I had to walk my 88 yo Mom to the store to swap SIM card.
2) Every clerk at every shop can do that so for a determined criminal it is possible to bribe or threaten one.
3) Virtual operators (MVNOs) usually do not have physical locations and there is a dozen of them.