I don't think never worry about it again is quit correct

  What if someone registers with my old number? 
  If someone were to register with your old number on a new phone, then they will have an empty message history. Your contacts will also be made aware of a safety number change if they start messaging with the old number.

https://support.signal.org/hc/en-us/articles/360007062012-Ne...