GHC is an entire library too. It has parsers, AST datatypes, desugarers, typecheckers for both Haskell and Core, multiple intermediate representation libraries, ... and the actual compiler is "just" an application that uses that library.

An excellent overview: http://www.aosabook.org/en/ghc.html