I can't even imagine not changes the locks on a house you just bought. (Unless, as in your case, locks aren't even desired.) Even if the previous owner had no malicious intent, they might have accidentally kept a key that someone else could get ahold of... Or they might have given out a key, etc. There's no good reason to avoid it, IMO.

As a middle point, I’ve been living at my house in the Midwest for 4 years now and while we have locks, I didn’t changed them out from the previous owners. I suppose they could have some extra keys but I trust that they don’t and won’t come into my house (and unless they’re playing a really long game, they haven’t, and the previous owners live fairly close as we got some letters from their child’s daycare accidentally sent to us).

It's also a fun thing to do.

The re-key kit is only a few bucks, and it's a pretty simple process, and you get to enjoy the ingenuity contained in the lock while you work on it.