Yeah - At FastMail we default to only adding if it comes from someone in your addressbook. You can reduce it to just an addressbook group, or open to the world if you feel like living dangerously.