Ive done that too. But I've experienced a lot of people removing it again.

The result is uncomfortable, since Signal never knows the other side removed it, so messages land in thin air. Several times did I have awkward moments when someone was angry "I thought you'd let me know if it was cancelled" - "I did - I sent you a text about that two days ago! On Signal!"