Exactly this. I recently started a twitter for my academic career. Didn't share my contacts or anything (I only follow academic twitter too). I get tons of suggestions of people I know and several have followed me. The information is from their contact list because twitter knows my number and connected us. There's a clear benefit to this, but there's also privacy concerns too. The lack of control over this is what is concerning.