News for hackers. Stuff that matters.
Re:Data wrangling in Elixir with Explorer, the power of Rust... by losvedir on Friday April 14, 2023 @01:26PM (#35572142)
The video overview says the journey in data is "just starting". That's exciting! Any ideas or vision for where it's going in the future?

Also, I noticed in the demo that installing polars was very fast! Was the video trimmed, or was it cached or something? I remember when I last tried out polars in Elixir a while ago, it had to build the rust library and everything and took like 10 minutes.

Re:Data wrangling in Elixir with Explorer, the power of Rust... by josevalim on Friday April 14, 2023 @01:36PM (#35572279)
Things that are in our roadmap in relation to our data vision:

* Data management within your notebook (https://github.com/livebook-dev/livebook/issues/1604) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it

* We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by selecting a visualization: https://github.com/livebook-dev/livebook/issues/1545

We have other ideas, such as making SQL a more prominent citizen in Livebook and be able to build a custom canvas as your work on your notebook, but those will likely take longer to realize.

Re:Data wrangling in Elixir with Explorer, the power of Rust... by jasonpbecker on Friday April 14, 2023 @04:20PM (#35574228)
A vote here for a SQL cell. I want folks to use Livebook and Explorer more, but a very easy win for data folks who are not familiar with Ecto and are mostly writing complex select statements would be a sql code block that can easily reference a connection.

That would let people who are getting into Elixir for data work run a query, get an Explorer.DataFrame, and interact further that way.

Re:Data wrangling in Elixir with Explorer, the power of Rust... by josevalim on Friday April 14, 2023 @04:49PM (#35574555)
Today we already have the SQL smart cell that works on a connection. But it is isolated from the data frame world. We hope to find ways to better integrate these two. :)
Re:Data wrangling in Elixir with Explorer, the power of Rust... by hugobarauna on Friday April 14, 2023 @04:53PM (#35574599)
Good feedback, thanks.

Livebook already has a SQL Smart cell (https://livebook.dev/integrations/sql). It doesn't integrate with Explorer yet, but it's already possible to reference a database connection, run a SQL query, and visualize the results in a table.

Here's a video showing how to do that https://www.youtube.com/watch?v=F98OWdigCjY

Re:Data wrangling in Elixir with Explorer, the power of Rust... by cigrainger on Friday April 14, 2023 @01:29PM (#35572181)
We use the excellent Rustler Precompiled [1] library now so prebuilt binaries ship with the Elixir package. No Rust toolchain needed. :)

[1] https://github.com/philss/rustler_precompiled