The users it finds aren't necessarily ones you've interacted with -- I know from running single server instances
I stopped running an instance because of this functionality. I was running Mastodon on my home Internet and quickly became concerned about the image content it was downloading from users I hadn't interacted with, while I wasn't monitoring the server
There is a LOT of lolicon content on the Fediverse and NOTHING is worth the risk of running a server that might download material that's considered to be CSAM to my local computer while it is unsupervised
So you have to worry about both the quantity of content it downloads and what that content is
As an admin: no thanks.
Your statement is a hint to the problem.
From what I find on the Internet, you can still communicate/follow people on other servers without getting their timelines. It's just that the default setting is to pull everything from those relays and you need to disable it.
With that setting, you'll get everything people you follow on other instances, but not more. The "Federated timeline" tab will remain empty.
If you have a user that starts following 10k user in a short time, you have a spam bot and not a user.
The fediverse is not made in such a way that your social circle grows quickly. You usually starts by following people on your own instance, then people whose content is boosted by them and it only happens slowly and gradually.