You're right, of course, but it's not binary (either you're attached or you're detached). Instead, the idea is to become attached to subtler (read: causing more lasting happiness) aspects of life, as a consequence of which one becomes detached from the grosser (read: causing more fleeting happiness) aspects. This moving in stages applies on physical, emotional and intellectual levels.

Jumping to absolute detachment is not really possible.