Right there with you. I had assumed it would just look for folks on the network and give me the option at that point on who to add. Nope. I had folks replying back to me declining invite. I had people in my address book I never wanted to contact again. I finally got LinkedIn to stop retrying people from my list, but I failed to get them to forget my list apparently because, like you, some of these people/addresses are on my recommendation list.