Actually, I doubt it will take you all that long. At one point, I was following ~1,900 people; I unfollowed all of them to get back to the essence of things… since then, I’ve climbed back to 600, and I don’t believe it took me more than, hm… a few months? If you’re trying to re-amass such a following, instead of trying to keep it down (as I was), you shouldn’t have any trouble at all :D