News for hackers. Stuff that matters.
Re:GitHub and Jupyter IPython Notebooks by RogerL on Thursday May 07, 2015 @09:38PM (#9509542)
I am utterly giddy about this. I've spent a lot of time in the book I am writing to generate PDFs, to create TOCs for links to nbviewer, managing all these links and production every time I made a change. Now people can just view it directly in github. Life is as it should be.

There is just so much innovation and hard work coming out of this group, which is quite tiny. Any time I've raised an issue, even if it turned out to be my own dumb fault, gets the immediate attention and support of a core developer.

If you want to communicate with programmers there really isn't a better platform out there. The old 'write code, run code, save results, write latex, gen document, find error, repeat, opps code is out of sync with results....' is pining for the fjords.

Re:GitHub and Jupyter IPython Notebooks by jlrk on Thursday May 07, 2015 @04:18PM (#9507967)
Yes! A million times, yes! This lowers the barrier to sharing research results stored in a private repo with collaborators. Also, for those who have been looking for ways to display equations in markdown[1], you can embed the equations in markdown (Ipython uses Mathjax) within the notebook.

[1] https://github.com/github/markup/issues/274

Re:GitHub and Jupyter IPython Notebooks by mvermaat on Thursday May 07, 2015 @05:51PM (#9508570)
This is great! We use IPython Notebook a lot in our group, for exploratory data analysis, teaching, sharing experimental results, and as an electronic lab notebook. Inline GitHub rendering is very nice for all our public work.

If you run a private GitLab server, you could have a look at my patch which adds similar functionality to GitLab (in a rather ad-hoc way): https://gist.github.com/martijnvermaat/6926070

Re:GitHub and Jupyter IPython Notebooks by sytse on Friday May 08, 2015 @03:36AM (#9510417)
GitLab CEO here, thanks for maintaining your patch for GitLab. If you want to discuss contributing jupyter support to GitLab itself I'd be happy to discuss.
Re:GitHub and Jupyter IPython Notebooks by Myrmornis on Thursday May 07, 2015 @09:45PM (#9509565)
This is amazing. I think ipython notebook is fantastic in all regards. However, I feel there's a bit of an elephant in the room. The main tools I use as a programmer are shell and a text editor. I'm not about to start writing code in a web browser[1]. Is there any hope that there will be a good workflow for somehow editing .ipynb from a text editor or is that antithetical to the design? Could markdown perhaps be used as a primary editing format to target ipynb?

Looks like there's at least one project doing that. https://github.com/aaren/notedown

And there's the emacs-specific project https://github.com/tkf/emacs-ipython-notebook

[1] EDIT: I mean, not right now anyway. I've got a feeling that statement could seem a bit dated later in my life...

Re:GitHub and Jupyter IPython Notebooks by idunning on Thursday May 07, 2015 @09:56PM (#9509610)
It is an interesting point. In my usage, I don't view it as a one-or-the-other proposition. I use Julia, so if I'm doing some exploratory work and experimentation, or making something to present results, then the IJulia notebook is great. If I'm writing some serious longer-running stuff, or a package, I'm in an editor. Sometimes I'll write code in a separate file and call it from a notebook just to keep the notebook focussed on communicating something, and "hiding" the details.
Re:GitHub and Jupyter IPython Notebooks by ngoldbaum on Thursday May 07, 2015 @10:20PM (#9509676)
Yeah, notebooks are great for experimentation but not so much for anything that needs to be reused.

I tend to write prototype code in the notebook then copy/paste it into a package as I become satisfied that the code is correct.

Re:GitHub and Jupyter IPython Notebooks by aaren on Friday May 08, 2015 @05:19AM (#9510686)
I wrote notedown precisely so that I could edit notebooks in my text editor and version control them as plain text.

It would be good to be able to transparently edit a .ipynb in a text editor - I've been meaning to wrap notedown in a vim plugin but haven't had time. This still leaves the version control problem though.

Note that I recently enabled the reverse (editing markdown in the browser as if it was a notebook), enabled by setting the following in ~/.ipython/profile_default/ipython_notebook_config.py (or similar):

    c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'
This is useful for more interactive work, e.g. iterating on plots, whilst still having everything stored in markdown. See [1] for more info.

[1]: https://github.com/aaren/notedown/issues/22

Re:GitHub and Jupyter IPython Notebooks by julienchastang on Friday May 08, 2015 @01:35AM (#9510121)
I've been a big fan of TFK's EIN. Unfortunately, TFK no longer maintains the project and I have not had good luck with the various EIN forks. (Anyone else having better experiences?) I still get by with the original EIN by using older versions of ipython/IPyNB. Moreover, the IPyNB technology is getting somewhat bloated with widgets and whatnot and I don't know how that will integrate with an emacs environment. Nevertheless, I still hope emacs modes similar to EIN will be forthcoming in this area. I was hoping to propose a code sprint on this topic at SciPy this year, but I won't be going to Austin, alas.
Re:GitHub and Jupyter IPython Notebooks by wodenokoto on Thursday May 07, 2015 @10:36PM (#9509702)
Re:GitHub and Jupyter IPython Notebooks by munro on Thursday May 07, 2015 @06:04PM (#9508659)
So weird, I was playing with IPython Notebook with PyCharm yesterday. Then I went on to think about how it could be integrated into Github without building the compiled files into the repo. :)

Really there's no solution for extending Github except wait for them to integrate hooks. It would be nice if they made this more generic, and created a file render hook, that would allowed devs to render whatever types of files they want into Github Markdown.

Re:GitHub and Jupyter IPython Notebooks by jszymborski on Thursday May 07, 2015 @09:25PM (#9509506)
For the first time today I put an IPython notebook into a github repo, and thought "cool, they render on GitHub", only to read that they rolled it out today on HN.

Super neat.

Re:GitHub and Jupyter IPython Notebooks by hcrisp on Thursday May 07, 2015 @07:54PM (#9509195)
How does iPython integrate with PyCharm?
Re:GitHub and Jupyter IPython Notebooks by usef- on Thursday May 07, 2015 @08:27PM (#9509327)
PyCharm allows you to edit and run ipynb files directly – see here: http://blog.jetbrains.com/pycharm/2014/12/feature-spotlight-...
Re:GitHub and Jupyter IPython Notebooks by bigredhdl on Thursday May 07, 2015 @05:51PM (#9508569)
Just a general comment, but in my opinion IPython/Jupyter is worth learning Python for. It is such a great platform for tinkering with data or really any idea you might play with in code.
Re:GitHub and Jupyter IPython Notebooks by RogerL on Thursday May 07, 2015 @09:31PM (#9509524)
The notebooks support a very large number of kernels, including Julia, R, bash, Lua, Erlang, Perl, OCaml, and so many more[1].

[1] https://github.com/ipython/ipython/wiki/IPython-kernels-for-...

Re:GitHub and Jupyter IPython Notebooks by pash on Thursday May 07, 2015 @09:59PM (#9509623)
And in case you were wondering, GitHub does render notebooks using other language kernels. For example, here's a notebook in Haskell: https://github.com/gibiansky/IHaskell/blob/master/notebooks/...
Re:GitHub and Jupyter IPython Notebooks by baldfat on Friday May 08, 2015 @06:51AM (#9510862)
I would like to also make note you can use any of the other languages that Juypter supports. I am using R right now and it works, but it isn't as useful as RStudio.

I am guessing that the Juypter notebook format might get traction with other projects besides Jetbeans.

Re:GitHub and Jupyter IPython Notebooks by rcpt on Thursday May 07, 2015 @05:12PM (#9508336)
Hey this is great! I just took a look and saw that a notebook [1] I was tracking in github is now rendering beautifully - and I didn't have to do anything!

https://github.com/rcompton/ml_cheat_sheet/blob/master/super...

Re:GitHub and Jupyter IPython Notebooks by voltagex_ on Thursday May 07, 2015 @10:36PM (#9509699)
They've really relaxed the filesize restrictions for these - I don't think a 7mb PDF will render in GitHub but this notebook will.
Re:GitHub and Jupyter IPython Notebooks by batbomb on Thursday May 07, 2015 @05:32PM (#9508449)
There's a new problem which still hasn't been resolved w.r.t. iPython notebooks and reproducible results, and that's the problem of private, ephemeral, or mutable data. At the very least, iPython theoretically helps with the audit trail for private and mutable data, but it doesn't necessarily foster (re)producible results or alternative analysis.

This is a more general problem overall, but potentially something like magnet URIs and bittorrent could really help with part of the problem. (I don't really believe git as a system nor GitHub as a platform to be the appropriate place to solve this either).

Re:GitHub and Jupyter IPython Notebooks by tmarthal on Thursday May 07, 2015 @08:16PM (#9509288)
Are you saying that people aren't storing data on github? How large of datasets are you talking about?

It's a shame that Github's large file storage (LFS) mechanism isn't publicly released yet, because I am sure that will appease your problems. (This is assuming your datasets don't fit into a csv (note that the current hard limit for github files is 100MB, and a 100MB CSV file is a lot of data)).

Re:GitHub and Jupyter IPython Notebooks by Fomite on Thursday May 07, 2015 @10:04PM (#9509634)
It should be noted that there are reasons beyond file size not to store data on GitHub, though I do regularly generate simulation data that's way larger than 100 MB CSV files.
Re:GitHub and Jupyter IPython Notebooks by tmarthal on Friday May 08, 2015 @01:44AM (#9510156)
Can you list the reasons that you're thinking of not storing data on Github? I would assume that you have a (paid) user or organization account and keep the repo private, unless it was for a published paper.

I was thinking more along the lines of data that you'd want to graph or preset for analysis, as in a notebook or a paper. I would (hopefully) assume that your simulation data would be generated and used in such a way that it wouldn't need to be stored permanently. At least, when I was doing simulation based analysis, I wasn't necessarily concerned about any individual run, but rather a combination of a bunch of simulations (all of which were ephemeral).

Re:GitHub and Jupyter IPython Notebooks by Fomite on Friday May 08, 2015 @09:07AM (#9511420)
Because there are data use agreements for health data with personal identifiers, and the vast majority of them aren't going to accept "We're keeping a copy on a private Github repo".

Nor, to be blunt, should they.

My particular simulation work is rather interested both in individual runs (and indeed, individuals within those runs) as well as summarization.

Beyond that, what use is there to putting just the "summary" data online, when the underlying data made that still exists as "you're just going to have to trust me". Being able to replicate my figure code doesn't get people very far.

Re:GitHub and Jupyter IPython Notebooks by sciencerobot on Thursday May 07, 2015 @11:36PM (#9509839)
Just curious: why not just provide the code that generates the simulation data?
Re:GitHub and Jupyter IPython Notebooks by Fomite on Friday May 08, 2015 @09:11AM (#9511445)
Because it's an absolutely massive code base that requires access to some serious HPC resources, which basically makes it difficult if not impossible to reproduce for most people. Putting something out there like that also implies something of an obligation to maintain and support it, which isn't what we do.

There's also a weird data rabbit hole. Is the code that generates the simulation data enough, without the underlying data that code uses? Some of that is either protected or proprietary, so even with the code, it's utterly useless.

Re:GitHub and Jupyter IPython Notebooks by vhffm on Friday May 08, 2015 @04:06AM (#9510489)
Simulations may run for weeks or months on large computing clusters. People wanting the resulting data may not have suitable access and/or resource allocations to repeat the runs.
Re:GitHub and Jupyter IPython Notebooks by qznc on Friday May 08, 2015 @03:34AM (#9510413)
It helps with ephemeral data, unless I misunderstand you. Notebook document how to generate the ephemeral data.

Surely, it does not help with private or mutable data, but how would magnet URIs and bittorrent help? If it is private, it cannot be shared with bittorrent either and sharing mutable data via bittorrent is not its primary use case.

Bittorrent might help with big data, where big means 100MB+ in the case of Github. There are other approaches for big files in git (git annex, Github's LFS, git-bigfiles, etc).

Re:GitHub and Jupyter IPython Notebooks by Fomite on Thursday May 07, 2015 @06:45PM (#9508888)
This also means, at GitHub repos can get DOIs via FigShare, you can have DOIs for an iPython Notebook, which is awesome.
Re:GitHub and Jupyter IPython Notebooks by saboot on Thursday May 07, 2015 @04:23PM (#9508010)
Okay, what is the best format to store images in? My notebooks contain lots of graphs and output but I'd rather avoid saving the raw pixels. So far I've been using SVG with mixed success. Does anyone use anything else to save space?
Re:GitHub and Jupyter IPython Notebooks by rcpt on Friday May 08, 2015 @05:06PM (#9514155)
I just tried out https://pypi.python.org/pypi/ipynbcompress/0.3.0 and it worked pretty well. Need to use 'png' to get nice rendering on github.

The last cell here has my use https://github.com/rcompton/ml_cheat_sheet/blob/master/super...

Re:GitHub and Jupyter IPython Notebooks by stevetjoa on Friday May 08, 2015 @02:51AM (#9510325)
Great idea! Although it appears that the Audio widget does not display correctly. Compare:

http://nbviewer.ipython.org/github/stevetjoa/stanford-mir/bl...

https://github.com/stevetjoa/stanford-mir/blob/master/notebo...

Re:GitHub and Jupyter IPython Notebooks by huac on Thursday May 07, 2015 @07:20PM (#9509068)
Hope support for R-markdown is next!
Re:GitHub and Jupyter IPython Notebooks by haddr on Thursday May 07, 2015 @10:20PM (#9509671)
I was also a bit dissapointed for no mention about R :( Anyway, i think it's possible to get notebook for R using this: https://github.com/IRkernel/IRkernel

look at this demo: https://github.com/IRkernel/IRkernel/blob/master/Demo.ipynb

Re:GitHub and Jupyter IPython Notebooks by shepardrtc on Thursday May 07, 2015 @03:37PM (#9507699)
So they're only displaying the notebooks, but not actually allowing you to work in them, right? Because for a minute there I was pretty excited!
Re:GitHub and Jupyter IPython Notebooks by andmalc on Thursday May 07, 2015 @07:54PM (#9509193)
Excited about IPython notebooks generally or possibility of editing them on GitHub?

If you want one to edit, try here: https://cloud.sagemath.com

Re:GitHub and Jupyter IPython Notebooks by munro on Thursday May 07, 2015 @06:13PM (#9508704)
There are a lot of tools for editing notebooks, and they're complex. I also don't use their online code editor, because mine is better. Through my personal experiences, my assumption is they'll never take on that project.
Re:GitHub and Jupyter IPython Notebooks by carreau on Thursday May 07, 2015 @04:01PM (#9507863)
you cannot work on it yet.
Re:GitHub and Jupyter IPython Notebooks by rspeer on Thursday May 07, 2015 @04:19PM (#9507971)
It sounds like you expect this to change. If you could actually run the commands, where would the computing power come from?
Re:GitHub and Jupyter IPython Notebooks by takluyver on Thursday May 07, 2015 @04:27PM (#9508040)
If you go to https://try.jupyter.org/ , you'll get a temporary notebook server in a Docker container, courtesy of Rackspace. It's automatically destroyed after a few minutes of inactivity.
Re:GitHub and Jupyter IPython Notebooks by carreau on Thursday May 07, 2015 @06:14PM (#9508717)
editing does not either imply running code. Markdown cells could be easily edited without a running kernel.
Re:GitHub and Jupyter IPython Notebooks by cabhishek on Thursday May 07, 2015 @06:39PM (#9508855)
Very thoughtful and awesome. Now is there an easy way to save notebooks directly on GitHub?(git as storage system?). That would make the whole flow seamless and make sharing and versioning easy :)
Re:GitHub and Jupyter IPython Notebooks by yes_or_gnome on Thursday May 07, 2015 @03:54PM (#9507818)
Screen has both 'c' and 'C-c' set to new window by default. tmux only has 'c' by default.

From https://www.gnu.org/software/screen/manual/html_node/Default...:

  C-a c
  C-a C-c
  (screen)
  Create a new window with a shell and switch to that window. See Screen Command. 
Edit: d'oh. I had to reread the original comment. You already know this, but by adding 'C-c' I don't have to release the Ctrl button.

You can add this to tmux.conf to get screen-like behavior:

  # New Window
  bind C-c new-window
Re:GitHub and Jupyter IPython Notebooks by wyldfire on Thursday May 07, 2015 @05:14PM (#9508343)
The wires are crossed on HN or your tabs. Your tmux response is in the github-ipython notebook thread.
Re:GitHub and Jupyter IPython Notebooks by agrothberg on Friday May 08, 2015 @02:12AM (#9510215)
It would still be nice if IPython Notebook had an option to save the code separate from the output, for example as two files in the same directory. This would make source controlling easier.
Re:GitHub and Jupyter IPython Notebooks by c0lin on Friday May 08, 2015 @04:35AM (#9510559)
Running `ipython --notebook` with the additional option `--script` may be what you are looking for. This saves a `.py` file with the same name next to your `.ipynb` file (the non-code sections are present, but they are commented out).
Re:GitHub and Jupyter IPython Notebooks by vhost- on Friday May 08, 2015 @02:13AM (#9510223)
I just started using these for my data science work and it has been absolutely awesome. Even better now that they display on Github and apparently (based on a comment below) Gitlab too.
Re:GitHub and Jupyter IPython Notebooks by andersonfreitas on Friday May 08, 2015 @07:00PM (#9514607)
Would be nice to have .ipynb preview on Gist also!
Re:GitHub and Jupyter IPython Notebooks by calebm on Thursday May 07, 2015 @05:24PM (#9508402)
This is awesome! Similar to nbviewer.ipython.org functionality.
Re:GitHub and Jupyter IPython Notebooks by bruce487 on Friday May 08, 2015 @04:41AM (#9510571)
Awesome, thanks for sharing.