Take a look at Apache Merton's architecture. They seem to have dealt with a number of inherent issues in the problem putting together data processing pipelines of open source components and ensuring they work together. The project is active but incubating right now.