In everything except the most exceptional circumstances knowing what to build will be a blend of written requirements and shared knowledge within the team. You can advocate for comprehensive requirements which pushes the effort towards the written spec side, or you can advocate for "working software over comprehensive documentation" which pushes the effort towards using more shared knowledge. Depending on the nature of your project, one or the other might be preferable.

For most projects, both "knowing what to build" and "working software" will never be 100% clear.