We made the same mistake with Anthill Pro a long time ago. The plugins were of various quality and caused us a lot of headache.

Now our build and test suite is just a script each and it can pretty much on on any CI server.