News for hackers. Stuff that matters.
Re:The Future of Notebooks: Lessons from JupyterCon by zmmmmm on Saturday August 25, 2018 @01:21AM (#17839674)
This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.
Re:The Future of Notebooks: Lessons from JupyterCon by dangirsh on Saturday August 25, 2018 @03:07AM (#17839926)
ob-ipython enables IDE-like editing features within the code cells. It's embedded in a polyglot, git-friendly, literate programming environment called Org-mode. I use it every day and love it. Other goodies:

- easily manage multiple kernels (in different languages / machines) in one file

- tree-based organization manages complexity better than linear notebooks

- no browser in sight (unless you need interactive widgets)

- highly exportable, including to ipynb via ox-ipynb

Downsides:

- Emacs-only

- small user base / limited docs

- can't easily import from ipynb

- async cell execution support is early-stage

See Scimax ipython for examples

Re:The Future of Notebooks: Lessons from JupyterCon by peatmoss on Saturday August 25, 2018 @08:13AM (#17840537)
Org-mode is strictly speaking the most powerful notebook programming environment out there.

I’m also much more excited by R Notebooks as implemented by RStudio than I am Jupyter. R Notebooks take the same basic approach as org-mode, implementing a smaller set of functionality built around a more mainstream-palatable Markdown format.

I don’t like R Notebooks as much as org-mode (why reinvent the wheel!?), but at least the general approach plays nicely with git.

On the other hand, the lisp-addled part of me starts to think that the juypyter format being json might actually be a step up from a poorly specified markup format that has to be parsed into data structures.

Maybe the real issue is that git is an insufficient revision control system, and that we need revision control systems that can revision data structures, rather than simple text diffing.

Re:The Future of Notebooks: Lessons from JupyterCon by jaipilot747 on Saturday August 25, 2018 @09:50AM (#17840753)
Could you please elaborate on how you use orgmode for interactive programming?
Re:The Future of Notebooks: Lessons from JupyterCon by chimprich on Saturday August 25, 2018 @07:17AM (#17840409)
Interesting. I use emacs for hacking Python but I've never heard of this tool. Do you actually use this for software development or is it more of a data-science-type exploration tool?
Re:The Future of Notebooks: Lessons from JupyterCon by dangirsh on Sunday August 26, 2018 @12:10AM (#17844209)
I use Org for software development whenever I can, which is currently everyday.

Usually, new code starts in cells with some Org-managed context (e.g. a Jupyter kernel in a remote container with some DB/service access). This is done using the :session code cell keyword, which works per subtree. Managing remote sessions like this generally keeps me away from terminals.

Surrounding the cell are various mini-dashboards with useful docs / links / commands for that part of the project. Since Org supports embedding elisp and shell commands in clickable links [1], these mini-dashboards can be made very quickly.

Org lets me edit the code using the proper Emacs mode for its language, while pulling dynamic completion / docs from the Jupyter kernel. Just like Jupyter notebooks, I can view rich outputs from the cells in-line. I can then name the outputs and make them inputs to other cells, including ones in different languages / kernels. AFAIK that's an Org-only trick.

Most code eventually finds it's way to normal source files (see Org's "tangle" feature). This feels more natural than moving code from notebooks since, again, the cell editing mode is the same as the one for source files.

Org's tree-manipulation capabilities + support for multiple sessions means that (so far) I've only ever needed 1 Org file per project. I track this in git, which is simple since Org is just plain-text. To share with non-Org users, I usually export to ipynb [2] or, for static docs, HTML [3].

[1]: https://orgmode.org/manual/External-links.html [2]: https://github.com/jkitchin/ox-ipynb [3]: https://github.com/fniessen/org-html-themes

Re:The Future of Notebooks: Lessons from JupyterCon by chimprich on Monday September 03, 2018 @04:01AM (#17901155)
Thanks.
Re:The Future of Notebooks: Lessons from JupyterCon by d0mine on Saturday August 25, 2018 @11:00AM (#17841030)
related "literate devops in emacs" https://youtu.be/dljNabciEGg
Re:The Future of Notebooks: Lessons from JupyterCon by staticassertion on Saturday August 25, 2018 @01:33AM (#17839705)
What? What are you using Jupyter notebooks for where you want maintenance? They should be records of data analysis/ procedures, not code that runs in production or something.
Re:The Future of Notebooks: Lessons from JupyterCon by brianberns on Saturday August 25, 2018 @01:47AM (#17839736)
Did you read the article?

"A Netflix engineer described how they have replaced Bash scripts with Jupyter notebooks for ETL pipelines and cron jobs."

Re:The Future of Notebooks: Lessons from JupyterCon by laichzeit0 on Saturday August 25, 2018 @01:44AM (#17839731)
Maybe read the article. They are experimenting with putting the notebooks directly into production ala bash script. I don’t think this is a great idea either.
Re:The Future of Notebooks: Lessons from JupyterCon by staticassertion on Saturday August 25, 2018 @12:58PM (#17841541)
Yeah, I read it after, silly of me to comment first.
Re:The Future of Notebooks: Lessons from JupyterCon by ekianjo on Saturday August 25, 2018 @01:36AM (#17839716)
exactly. I use it for quick data exploration and experimentation but it remains at that level.
Re:The Future of Notebooks: Lessons from JupyterCon by gaius on Saturday August 25, 2018 @01:44AM (#17839730)
People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
Re:The Future of Notebooks: Lessons from JupyterCon by balaba on Saturday August 25, 2018 @07:30AM (#17840436)
everyone knows that excel is a drawing program https://www.thisiscolossal.com/2017/12/tatsuo-horiuchi-excel...
Re:The Future of Notebooks: Lessons from JupyterCon by kgwgk on Saturday August 25, 2018 @05:26AM (#17840214)
I can’t tell if this is sarcasm or not... I’ll assume it is and upvote :-)
Re:The Future of Notebooks: Lessons from JupyterCon by ekianjo on Saturday August 25, 2018 @02:57AM (#17839908)
Its not the fault of the tools if users dont know any better.
Re:The Future of Notebooks: Lessons from JupyterCon by fsloth on Saturday August 25, 2018 @06:15AM (#17840294)
So what would you suggest a financial modeler, who has no experience of any programming environments, to use instead?

The value of Excel is that is a zero-config tool, available everywhere as 'standard' business installation, allows very quick iteration with visual output in certain range of tasks, is battle tested in millions of computers... etc. And everyone else is using it too.

For a programmer it's easy to suggest to just pick some good language. Once you know two programming languages you can navigate pretty soon fluently in third.

But picking up the first language fluency? That's hard. You can't just suggest to use python or such. You need to provide a tool that holds hands, has lots tutorials available etc.

Visual programming is not necessary the answer either. I've understood that without discipline LabView programs become actual visual spaghetti pretty quickly.

Re:The Future of Notebooks: Lessons from JupyterCon by ekianjo on Sunday August 26, 2018 @12:26AM (#17844252)
of course its hard. Just like its hard a new language thats is far from your native tongue. But we should not be scared of learning something difficult, because the returns are well worth the investment. There is only so much you can do in Excel and if you want to advance your career you need to move past it even if you have no programming background. Additionally, its never been easier than in 2018 to find free resources to learn. The only things you need are awareness, time, and will.
Re:The Future of Notebooks: Lessons from JupyterCon by TeMPOraL on Saturday August 25, 2018 @05:15AM (#17840200)
It's also not the fault of the tools if there are no better alternatives.
Re:The Future of Notebooks: Lessons from JupyterCon by stewbrew on Saturday August 25, 2018 @02:35AM (#17839852)
Some simpler notebook-like environments stay closer to source code in that they basically are source code with interleaved results (e.g. as comments). IMHO they hit a sweet spot between REPLs and those notebook environments inspired by mathematica, maple and similar more mathematically oriented software products.
Re:The Future of Notebooks: Lessons from JupyterCon by TeMPOraL on Saturday August 25, 2018 @05:11AM (#17840188)
That's just a REPL with basic editor integration (eval at point, paste result). Surprisingly unpopular outside Emacs/Lisp land.
Re:The Future of Notebooks: Lessons from JupyterCon by stewbrew on Saturday August 25, 2018 @08:09AM (#17840529)
Well yes, but these notebooks are little more, aren't they.