News for hackers. Stuff that matters.
Re:Explorer – A library to bring dataframes to Elixir by jasonpbecker on Saturday April 30, 2022 @12:17PM (#31217090)
Flow/Broadway + how nice Ecto is already had me thinking, "Elixir is super interesting for data engineering." Explorer and Livebook have me thinking, "Elixir has the best shot at unseating Python."
Re:Explorer – A library to bring dataframes to Elixir by 3jckd on Saturday April 30, 2022 @12:38PM (#31217230)
It's interesting what/when/if will unsettle Python. And what the adoption would be like.

Julia has been <designed> to unsettle Python in the data space but to no avail.

¯\_(ツ)_/¯

Re:Explorer – A library to bring dataframes to Elixir by jasonpbecker on Saturday April 30, 2022 @02:26PM (#31218053)
Julia was designed to unsettle a creeping set of python in the domain of Perl, C++, and Matlab in scientific computing and various quant analysis that’s pretty distant, IMO, from what industry data engineering, ML, and analytics work was doing.
Re:Explorer – A library to bring dataframes to Elixir by rch on Saturday April 30, 2022 @04:38PM (#31219095)
Factor in Rust integration via Rustler and it starts to look like a 'when not if' dynamic.
Re:Explorer – A library to bring dataframes to Elixir by sergiomattei on Saturday April 30, 2022 @03:56PM (#31218764)
Doubt it. Elixir is a leaky abstraction over Erlang, so OTP concepts are a huge barrier of entry for non-software engineers.

e.g. to this day starting a small library requires learning supervisors, genservers, etc. Unless you’re doing a one off .exs, you’re going to be really confused by what “mix new” does.

Saying this as a passionate Elixirist — there’s nothing wrong with the leaky abstraction part, it’s what makes it powerful — but dethroning Python isn’t happening anytime soon.

Re:Explorer – A library to bring dataframes to Elixir by josevalim on Saturday April 30, 2022 @04:16PM (#31218896)
I am not sure I agree with the OTP bits.

We often tend to compare having to learn OTP with “not having to learn OTP” but the reality is that we need to compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes).

Of course, one can say: “I wouldn’t use threads in other languages” but that’s potentially a bigger problem, as you end up with software which can’t easily exploit concurrency (which is especially important in the data domain).

I still think what you said has some truth to it, but I think the frame of reference is a bit off. I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :)

PS: “mix new” does not generate supervision trees by default.

Re:Explorer – A library to bring dataframes to Elixir by sergiomattei on Saturday April 30, 2022 @05:17PM (#31219455)
> compare OTP with concurrency tools in other languages (e.g. threads, locks, mutexes)

That’s a very fair point, but is the average data scientist going to be learning this or offloading that task to a software engineer?

> I would rather learn a bit of OTP to get concurrency and fault tolerance rather than learn nothing and get none of it. :)

Agreed —- I’ve been using Elixir for a bit and am only now learning about supervision trees and genservers. It’s such an exciting part of the language.

> PS: “mix new” does not generate supervision trees by default.

Yup and that was confusing for me as a beginner — I recall being confused over where the entrypoint of the app was. Coming from a language like Python.

Hi José, huge fan. Thanks for dropping by this thread!

Re:Explorer – A library to bring dataframes to Elixir by ravi-delia on Saturday April 30, 2022 @05:53PM (#31219748)
Maybe I've just forgotten what learning Elixir was like, but I feel like OTP never gave me all that much trouble? I guess it might be a little weird, but conceptually it's not that different from how objects work and presumably Python data people do fine with that. I feel like the sticking point might be the syntax. Elixir has a very developer-friendly syntax, but the flexibility (all the ways to call functions, implicit returns, arity stuff, collection syntax, even macros) makes it way harder to just learn "incantations" and be able to read other people's code.
Re:Explorer – A library to bring dataframes to Elixir by tfp137 on Saturday April 30, 2022 @02:06PM (#31217890)
I thought it would be Clojure, but that was 10 years ago and Python is still (inexplicably?) dominant.