Your friend's Mac creating a knowledge graph is totally different because they control it. It gives them power to remember more about their interactions with you, but they would have to actively abuse it to hurt you in any way. If you can't trust someone with information then don't give it to them in the first place expecting them to forget it.

The problem is what happens if you get hacked or Apple decides to sell you out/work with law enforcement. I would only trust code like that if it was fully transparent about what it's doing, open source and strictly opt in. Apple is doing it in the background and that makes it instantly suspicious.