The first stage is the LLVM front end, Clang and other language parsers. They generate IR.
The middle stage is the LLVM optimizer, opt, which takes IR and runs a variety of optimization passes. Opt takes and gives IR.
http://llvm.org/docs/Passes.html
The third is the backend code generator. It lowers IR to the target ISA. There's a tablegen description, there's are MI scheduler models which abstract architectural information as well.
http://llvm.org/docs/WritingAnLLVMBackend.html
I think you really meant can you write an optimizing pass specifically for your target? Yes. But I don't think that modifying the real front end Clang is going to get you much of anything. Intrinsics are different; that's the programmer inserting ISA specific instructions rather than Clang extracting ISA specific information. Really, modifying Clang isn't going to get you anything.
Thomas, what were the 'small' Clang specific tweaks?