Hence why most meetings have "minutes" sent after them. I don't see what you're doing as any different, so good on you. Also, if someone gets angry about you "confirming" your discussion with them, well then, maybe you want to make sure everything you talk about with them is confirmed.