Sure, it'll impact you, but so will the loss of time, funds, etc., from attending to academic pursuits. Key is to find a good first job as soon as possible and leverage it to find your next job as soon as possible. After you're established a career path, the percent of opportunities you have access to will be less and less limited by your academic history. Generally speaking, my suggestion is to go to school; though that's not the same advice I'd give to my past myself either given the opportunity.