This reminds me of anecdotes from... Time Management for System Administrators, by Tom Limoncelli. He talks exactly about "Won't people think I don't care if I have to write it down in a diary?" and mentions how, at least in his experience when reaching for diary for everything, people actually reacted positively - the extra effort of ensuring a note was made, calendar appointment created, schedule arranged made them feel that he was willing to spend some effort on them, instead of just doing the minimum.