The fear is that you destroy one or more of your inhabited planet not just a lifeless ball. Which I would hope you and most people would consider a significant downside. That risk is balanced vs the benefit of sending self replicating probes to every star system which just doesn’t seem that useful.

Now is that a large or small risk? I don’t have the technology but it’s something that could reasonably concern a civilization.