It's unsafe. They could lock you out from your Signal account and impersonate you. Someone who does not know you changed your number, forgot about it or does not think about this could then send a message to the person who has your old phone number thinking it's you at the other end. Most people don't bother with the warning about the security number having changed. I also personally assume that the phone number of a contact in Signal is theirs and that I can send an SMS or call them with this phone number. This not being true is at best confusing even if you are not locked out.

It'd also be nice to let the current user of your old number to be able to join Signal without issues.

I'd say don't play with fire and migrate as soon as possible before having issues.

You probably could follow https://support.signal.org/hc/en-us/articles/360007062012-Ch...

Then the people with who you discussed on Signal will be notified of your number change. If you can't follow this, better decide to handle it the best way you can instead of having to deal with this when you have lost access.