News for hackers. Stuff that matters.
Re:Dataflow, a self-hosted Observable notebook editor by keeganj on Thursday May 13, 2021 @02:44PM (#27145719)
I'm not a data scientist, but I've been interested in the idea of a "code notebook" ever since Jupyter hit it big. I write mostly in JS/TS for application logic, so this looks like it could be really useful.

Related, does anyone have any recommendations of a (Postgres) SQL "notebook"? I don't really need any visualizations, more just a markdown integrated doc that allows me to lay out the different queries I use to answer a question.

Re:Dataflow, a self-hosted Observable notebook editor by javierluraschi on Thursday May 13, 2021 @03:16PM (#27146151)
For viz/DS/ML/AI with JS/TS is either observablehq or and IDE with custom extensions; this project looks relevant if you are already into observablehq.

Shameless plug, we are building a few tools for JS to narrow down this gap as well: - https://hal9.ai (Drag&Drop / IDE) - https://marketplace.visualstudio.com/items?itemName=Hal9.hal... (VSCode extension) - https://observablehq.com/@javierluraschi/running-nodejs-in-o... (ObservableHQ extension)

Would love to chat if you are interested in providing feedback, I'm in javier at hal9.ai. Cheers.

Re:Dataflow, a self-hosted Observable notebook editor by natrys on Thursday May 13, 2021 @04:40PM (#27147156)
Emacs and Org-mode has great integration with multiple SQL implementations including Postgres (via org-babel). Org-mode tables are pretty neat, and you can have query result directly populated into tables. Read this blogpost if you are interested:

https://fluca1978.github.io/2021/01/18/PostgreSQLLiteratePro...

Re:Dataflow, a self-hosted Observable notebook editor by Hasnep on Thursday May 13, 2021 @02:52PM (#27145816)
Rmarkdown notebooks can contain SQL chunks, so you'd only need to use R to configure the connection. [1]

[1] https://bookdown.org/yihui/rmarkdown/language-engines.html#s...

Re:Dataflow, a self-hosted Observable notebook editor by keeganj on Thursday May 13, 2021 @03:19PM (#27146181)
I didn't know you could write SQL directly in Rmarkdown like this, very interesting. Thanks!
Re:Dataflow, a self-hosted Observable notebook editor by pbowyer on Thursday May 13, 2021 @04:06PM (#27146730)
Same, when I've read the docs I've always got the impression that it was R only supported.
Re:Dataflow, a self-hosted Observable notebook editor by PuddleOfSausage on Friday May 14, 2021 @02:14AM (#27151270)
There are loads more supported via knitr. Scroll to the top of that linked page in this thread for the list.
Re:Dataflow, a self-hosted Observable notebook editor by amcaskill on Thursday May 13, 2021 @04:11PM (#27146779)
I am working on a SQL-in-markdown reporting tool called evidence.

It’s feels like a markdown doc that runs SQL.

https://evidence.dev/

Re:Dataflow, a self-hosted Observable notebook editor by keeganj on Thursday May 13, 2021 @05:06PM (#27147473)
This is almost exactly what I was imagining. Just subscribed to updates, very interested to see what this becomes!
Re:Dataflow, a self-hosted Observable notebook editor by simonw on Thursday May 13, 2021 @03:16PM (#27146149)
Weirdly my Django SQL Dashboard project may fit the bill a bit here: you can build up a "dashboard" (which is a tiny bit notebook-like if you squint at it the right way) with multiple SQL queries on it, and save that either as a bookmark or as a "saved dashboard" with a URL.

https://django-sql-dashboard.datasette.io/

In my own work I've been using it for the kind of things that I would normally use a Jupyter notebook for - gathering together research on problems I'm trying to solve.

Re:Dataflow, a self-hosted Observable notebook editor by keeganj on Thursday May 13, 2021 @03:27PM (#27146269)
Interesting take, I'm not deep in the python ecosystem, but this looks like it's lightweight enough to function as a refreshable notebook. Will give this a try, thanks!
Re:Dataflow, a self-hosted Observable notebook editor by qbasic_forever on Thursday May 13, 2021 @05:13PM (#27147549)
I like the ipython-sql magic in Jupyter: https://github.com/catherinedevlin/ipython-sql Depending on what you're doing you might be able to get away entirely with just using it and some basic queries, i.e. no python glue code in the notebook at all. But worst case you might need a cell to open up the DB connection and make the magic aware of it, then you can execute clean and simple SQL queries in cells using the magic.
Re:Dataflow, a self-hosted Observable notebook editor by thejosh on Thursday May 13, 2021 @09:18PM (#27149525)
Yeah ipython-sql is great and works well, and can use an environment variable for the connection string.
Re:Dataflow, a self-hosted Observable notebook editor by tlarkworthy on Thursday May 13, 2021 @05:02PM (#27147428)
Re:Dataflow, a self-hosted Observable notebook editor by okennedy on Thursday May 13, 2021 @03:45PM (#27146444)
It's based on Spark rather than Postgresql directly, but I'm part of an effort to build a workflow system disguised as a notebook callled Vizier [1]. SQL is a first-class primitive in Vizier, and the notebook plays nice with postgres (you can load from and unload to postgres using Spark's native data loader).

[1] https://vizierdb.info

Re:Dataflow, a self-hosted Observable notebook editor by RocketSyntax on Thursday May 13, 2021 @03:14PM (#27146111)
Lots of jupyter magic `%` commands for that already https://www.datacamp.com/community/tutorials/sql-interface-w...
Re:Dataflow, a self-hosted Observable notebook editor by sixdimensional on Thursday May 13, 2021 @05:45PM (#27147890)
Apache Zeppelin is one open source option - https://zeppelin.apache.org.
Re:Dataflow, a self-hosted Observable notebook editor by gradys on Thursday May 13, 2021 @02:50PM (#27145798)
Maybe just a Python notebook with a Postgres client library and some helper functions to keep the amount of Python in the main body to a minimum?
Re:Dataflow, a self-hosted Observable notebook editor by shapiromatron on Thursday May 13, 2021 @03:20PM (#27146188)
re: sql notebook, this came up a few months ago and worked great when I played around with it: https://blog.jupyter.org/an-sql-solution-for-jupyter-ef4a00a.... It's just a different kernel you can install to an existing jupyter instance.
Re:Dataflow, a self-hosted Observable notebook editor by robertlacok on Thursday May 13, 2021 @06:32PM (#27148329)
Deepnote has native Postgres cells :) you can mix them with Python too.

Disclaimer - I work there :)

Re:Dataflow, a self-hosted Observable notebook editor by Siira on Thursday May 13, 2021 @06:19PM (#27148205)
org-babel should fit the bill.