Re:Show HN: Arc, an open-source Databricks alternative
by xupybd on Thursday March 25, 2021 @02:50AM
(#26577189)
I like the look of this but worry about adopting something as big as this. That said things tend to grow then I wish I'd started with something like this.
[ Parent
| on HN ]
Re:Show HN: Arc, an open-source Databricks alternative
by seddonm1 on Thursday March 25, 2021 @03:01AM
(#26577225)
A completely valid concern.
You can see that all stages in the video implement the PipelineStagePlugin: https://arc.tripl.ai/plugins/#pipeline-stage-plugins. This means you can safely remove them from the code base and recompile without that stage at all. These are all dynamically loaded at runtime so it should be easy (and to implement your own custom logic).
Similarly the Dockerfile https://github.com/tripl-ai/docker/blob/master/arc/Dockerfil... just includes the relevant plugins (if not in the main Arc repository) so you can easily remove them or the Cloud SDKs/JDBC drivers to reduce your surface area.
We have endeavoured to write a large number of tests but there is always room to add more.