By the time you notice and can react it's too late. There have also been many prominent examples of people who got their cryptocurrency exchange accounts broken into with SIM hijacking which was conducted while the victim was asleep.