News for hackers. Stuff that matters.
Re:Show HN: Bistro – A light-weight column-oriented data pro... by munro on Wednesday January 17, 2018 @02:10PM (#16170771)
I'm working on a columnar Python DSL right now, I think of it like SQLAlchemy for Pandas/Spark/Flink. My goal is to create a language that makes the cumbersome parts of the PySpark API much easier to express. I started off with the intent of replicating the R's dataframe API because it feels more fluid--but what you're doing feels eerie, because I came to the same conclusions as you around focusing on a language for columnar manipulations, and letting the "linking" become implicit. Then I want to transition to rethinking the ML pipeline, I like Spark's more than sklearn's, but there are still cumbersome parts that my intuition says can be solved by a columnar API.

So if I were you, I would do Python DSL, because it addresses my immediate needs of increasing productivity. :)

There are no comments on this comment yet.