In my experience (as a technical writer) the best documentation approaches it from both angles. The top-down approach connects all the pieces together, as you mentioned, and the bottom-up approach gives the nitty-gritty details that implementers need. That's why conceptual overviews and references are 2 of the most common content types.

See also this previous comment of mine about a common pitfall of reference documentation: https://news.ycombinator.com/item?id=20473713