At the directory level, I look at the history of commit messages to get a sense for how the module(s) under that directory have evolved over time. At the file level, I look at the history of commit messages and often dig into the associated diff in order to see how the API and/or implementation have evolved. I'll often follow the file-level diff to the full commit to see what the entire change entailed. This gives me tons of context on the what and the why of a codebase's evolution, and it's all aided by good commit messages.
Line level blame/annotate is sometimes useful when I'm trying to trace how some particular implementation came to be, but it is easy to lose the thread across deletions and renames, which I find file and directory level history to be much more resilient to.