Having never read an HR for Dummies book, and as a pure technical interviewer, I still frequently ask this question. The point is to not think of it as an adversarial question, like I'm going to mark you off for a bad answer.
It's simply to know how you want your career to proceed. Do you want to be an expert at some specific part of the programming world, an experienced generalist (consultant?), an architect, a manager or product lead, start your own startup, ... ?
This is a very important question for you and your manager to be on the same page about. It helps your manager know how to motivate you and what kinds of work will excite you to move your career forward. And it helps you evaluate via small experiences whether your vision matches up with your skills and what you actually enjoy.
So honestly, the question is: would you want to work for a company that wants you to lie to them? or a company that wants you to grow and roots for you when you do?
I doubt most of interviewers who ask the cliche setup* do it because they hope the best for you and are glad they found the perfect match, probably they just brought it from their previous experience with job seeking, or have read them in an average HR manual.
*
- what was your biggest success in your field?
- what are your weaknesses?
- where do you see yourself in 5 years?
A long time ago a kind of douchebag asked me them while clicking checkboxes on the screen after every answer. Needless to say I had no motivation to even care to go further in the process. A few years later I've heard from girlfriends just after graduation (non-technical field), they got them on every single one of their interviews, mostly asked by HR managers with no particular work experience. The first one is exceptionally funny when the interviewee is a post-grad.
I answered this way in a recent interview (and got the job). The company owners response was "I think working with us will help you learn a lot about starting your own company." At that point I really wanted to work with them.
And to be honest: how many companies out there want someone who tells them right away that experience and money obtained during their job is only to quit on the first moment they feel it's enough to run something on their own?
Lots of people say it, few do it. It also depends on the job. A job at a small company/startup needs self starters, jack of all trades kind of people. These same people are often the ones who want to do their own thing one day.
Many of these open ended questions came about from large companies hiring physiologist to help build and identify a profile of an individual that the company would like the hire. Organizations such as the FBI use similar strategies as well. The problem is that many smaller organizations that did not want to spend the money or simply did not know to spend the money on professionals started to emulate the larger companies, but they did not posses the answer keys once they had the answers they just left it up to the interviewer to divine whether it was the right answer or not. Read that last sentence again and then read the definition of a Cargo Cult. you will laugh at the mental image I promise. You see, these interviewing practices are simply the same mentality that created Cargo Cults, somewhere along the way they lost the intelligence and are just emulating the process in ritual and then divining the results.
And I still got the job.
On the flip side, I just hired a new finance person. I wanted someone who can eventually become a controller then CFO. I have no idea if they are really capable of it, but I want them to at least want it and to have demonstrated some activities to have tried working towards it.
What do I say? If I reach to high I endanger my own prospects at being hired. A buddy of mine answered the "see yourself in 5 years" question with "Probably working at Google". Needless to say he wasn't hired.
If I reach too low I sound like a deadbeat. If I answer with something like "Working at the position you are hiring me for" I sound like I have no vision and no drive.
There isn't a right answer to this question. Because the only right answer is "cliche".
The question is a test for you, but the answer you give, is also a test for the employer.
These 'where do you see yourself in 5 years' questions sound like straight from a 'HR for dummies' book. I would really appreciate if an interviewer reading this post explained me what such cliches are trying to test, other than BS skills. Cliche question => cliche response.