Show HN: Arc, an open-source Databricks alternative

Re:Show HN: Arc, an open-source Databricks alternative by lordgroff on Wednesday March 24, 2021 @08:34PM (#26575044)

I'm in the process of doing something like this internally, at a smaller scale, and it's interesting to see that many of the concepts I've been experimenting with and playing around with are formalized here in a similar manner. My "solution" doesn't build on Spark, as I just don't have enough data to necessitate it. I think the big difference is really the project's SQL first approach, which is probably going to polarize: personally, it's a decision I can't abide by, but I'm sure a lot of people will love that.

[ Parent | on HN ]

Re:Show HN: Arc, an open-source Databricks alternative by seddonm1 on Wednesday March 24, 2021 @08:43PM (#26575133)

Cool!

Most of the time we run Spark as a single-node (i.e. --master local[*]) as now we can easily utilise large nodes like 128 core, 512GB and Spark does scale vertically well but also runs relatively well on a small node like the Docker example on the website running on a laptop. The ability to run SQL against separate storage is Spark's killer feature in my view.

Arc does support the full Scala API which you can implement as a plugin (https://arc.tripl.ai/plugins/) so for advanced teams they have full control.

The reason we went for SQL-first is that we are trying to find the balance that allows Business Analysts to develop their own logic without having to learn Scala or even Python - as they probably already know SQL.

Hopefully some of the ideas are relevant to what you are building.

[ Parent | on HN ]

Re:Show HN: Arc, an open-source Databricks alternative by lordgroff on Wednesday March 24, 2021 @09:01PM (#26575278)

I'm reading the docs thoroughly, many excellent ideas, and I'm sure I'll be borrowing some concepts. I also want to commend you for putting in the time in developing proper documentation, always greatly appreciated.

[ Parent | on HN ]

Re:Show HN: Arc, an open-source Databricks alternative by seddonm1 on Wednesday March 24, 2021 @09:06PM (#26575326)

Great! I am glad someone is reading them as it has taken a huge amount of effort.

We also have a link from the Jupyter Completer (autocomplete) to the docs.

[ Parent | on HN ]