I was working on a small program that originally used the C# interop and found the latency pretty bad as well. For a small document, it was taking like 4 or 5 seconds to do what we wanted (dont remember exactly what that was atm). Eventually, I used some OpenXML library Microsft offered that was basically LINQ to XML with some types / extensions and found something interesting:

On smaller documents, the OpenXML manipulation blew the COM interop out of the water in terms of speed, but with larger documents (hundreds of thousands of rows), I think the COM interop was either faster or at least just as fast.