You need someone on your team who figures out what is important for the users and prioritizes accordingly. The arrangement where the person who does this also codes a bit can work (especially when the team is small and the application domain is highly technical), but it doesn't happen automatically after you've banished all "pointy-haired bosses" from the project nor is anyone who is a good engineer automatically good at this thing also.