This probably depends on the context.

If you send a friend an email, they may add you to their address book and set a reminder for your birthday while they're at it. They don't need your permission to do that, so why would a device acting on their behalf?

On the other hand, if a customer of mine asks for the data I have on them, do I have to send them my Personal Knowledge Graph facts about them?