Strikes me that the only difference between the way that you and I dress is that I wear a jacket ;-) (I can't stand ties). I wear the jacket because the silhouette is flattering, and there are all those handy pockets. Why should such a simple, very practical item make you think at such a person is over-dressing?
Anyway, that's literally what I wear to work most days. I have 3-4 pairs of dress slacks from a good store like those, different colors - black, charcoal gray, dark brown and a light gray (almost a tweed). Then you can buy a variety of different colored shirts. Then you just need a pair (or 2) of black and brown shoes, and matching belts (though you can probably get away with black shoes, and a black belt with the brown slacks, but I personally like to really look 'compete').
It's funny but I was starting to think about creating a post or blog entry on this because it's not that hard to dress really sharp without going too far. Also, if you aren't good at picking clothing bring a girl with you, or just ask one of the store clerks, they are usually really into this stuff and can point you in the right direction (but always pick things that appeal to you in some way, allowing them to influence your choice, not make it for you).