I worked for Microsoft doing VS integration for XNA Game Studio. As someone who had access to both the VS source and several VS internals experts, I too was kinda defeated by the VS extension API.

Good to know it's not just me, then :-)

I was all enthused when John Lam did a demo of scripting Visual Studio with IronRuby (I don't think it was his work, he was just demoing it), so I sent him an email asking after the source, but it turned out not to be something they were going to release. Which was a shame. He did make it look awfully slick.

Doesn't VS2010 have a new extension API?

Kinda.

VS2010 has a new, WPF-based shell/editor. That brings along many new .NET-based integration points and mostly obsoletes the old editor APIs. They've also improved the plugin-loading architecture and several other things. However, do do any meaningful integration, you still need to code to the poorly documented COM APIs. The project system, for example, hasn't changed much at all since VS 6.