I'm 6'2, 175, with a lean somewhat muscular build, more in the core and legs than in the arms.

(Wow, this is starting to sound like a personal ad, heh)

Anyway, if you're not "nearly that heavy" then I imagine you are quite thin? So yeah you don't need to be dropping weight! Beef it up!