I do the same but without writing down anything. I just keep it in my head. I always joke with how creepy I feel when I coincidentally meet someone for the second time after months, at a party or something and start to bring up details about that person into the conversation we are having. For now, everyone has been happy and kinda flattered that a pretty much unknown guy is able to remember precise details about their lives but I feel like a psycho.