http://zmjones.com/make/ http://blog.kaggle.com/2012/10/15/make-for-data-scientists/
I don't think serialization is a problem either. It is possible to make data structures that always exist as a single span of memory, so that they don't have a separate serialized representation. Most data structures are better served like this anyway, since prefetching and linear memory access is a huge speedup over pointer chasing. Building a data structure out of many separate heap allocations will also slow it down.
I'll often write the code so that each function is a bunch of value definitions in a pipeline, followed by a trivial return statement. No side effects if I can help it. Then I run it in the IntelliJ debugger and set a breakpoint on each return statement. When it hits the breakpoint, I instantly know the value of each subexpression, and can inspect them in case anything is amiss. If I need to rewrite something, I'll write the new code in a watch and immediately evaluate it. I basically use watches in the debugger as a REPL, except that I have the ability to factor out subexpressions and independently check the correctness of them, plus IntelliJ syntax-checks and autocompletes my code within the watch. Also, the sequence in which breakpoints are hit gives me a trace of the code.
The one downside is that I still have to recompile and re-run once I've fixed the bug and identified the correct code, but I mitigate that by architecting the app so that I can easily run it on a subset of data. So for example, my initial seed data set consists of a Hadoop job over a corpus of 35,000 1G files, but I wrote a little shim that can run the mapper & reducer locally, without starting a Hadoop cluster, over a single shard. And then I've got a command-line arg in that shim that lets me run it on just a single record within that shard, skipping over the rest. And I've got other flags that let me select which specific algorithm to run over that single record in that single shard in that 35TB 3B record corpus. I also rely heavily on crawling, but I save the output of each crawl as a WARC file that can be processed in a similar way, without needing to go back out to the Internet. The upshot is that within about 10 seconds I can stop on a breakpoint in any piece of the code, on any input data, and view the complete state of the program.
Like “this is my global state” - the entire state of the python process, all objects, etc.
Then I change a variable and that creates a new state. We have this now.
What I want is a rewind button, so I can go back to the previous state (quickly preferably) like a time travelling debugger.
Someone build this, I’ll pay for it (and I’m sure others would too)
I know there’d be some problems with external changes (API calls altering external system, writing to files, etc) that a time travelling debugger might not be able to reverse, but even so, I would live with this limitation happily if such a tool existed
Someone else in this thread mentioned Smalltalk: this is exactly what a Smalltalk image gives you.
The image is everything, all your objects, code, temporary, everything. And at any time you can do "save" or even "save as" to persist a copy.
And there are variants that can work with Python code as well.
I wonder what software engineering would look like if all the decades-long advancement of tools for stabilization and deployment of code had been researched over platforms that combined system and application code, like Smalltalk did. Instead, we consider both layers as isolated from each other. A lot of work over virtualization and containers seem to be used as a workaround for not having this integrated approach as part of the platform design.
It is definitely an approach more apt to personal computing than to industrial engineering, at least with the level of maturity of the tools available today. The increased ease of use of these environments may be a nice-to-have for professional developers, but they are essential for non-developers.
this is really cool, I'm going to give deeper into the code tonight. Awesome work
But what you describe is normal behaviour in any REPL. The main advantage notebooks have over shells is a better sessionhandling and richful datastructures. Images, markdown and stuff is a bit hard to display on a regular terminal. Ipython has the qt-frontend for this, but then moved to notebooks
I’ve been saying it for years, even before the great meme presentation this year: notebooks are not the appropriate medium for 99% of rapid prototyping or shared experiment work and notebooks make for terrible units of reproducibility (especially compared to just traditionally version controlled source code with a Makefile or with a Docker container etc).
Pedagogy is probably the only serious use case of a notebook. If you’re not working on a 100% pure pedagogy task, it’s a giant mistake to be doing what you’re doing in a Jupyter notebook.
- I work mostly from a clean, normal Python codebase in my normal IDE
- I'll have a file like "common.py" or something
- I keep a Jupyter notebook running always, and I'll do a (from importlib import reload) reload(common) to update any code in my live environment
I also:
- Cache a lot to disk (np.save/np.load).
- For every long-running method, have a tiny subset to unit test it
Java in Eclipse works exactly like this.
(just want to say: this concept is around for ages and Jupyter could adapt some of Matlab's GUI features for sure)
I guess that's the main reason why so many software developers don't "get" why people want to write code in a browser. Even normal Python developers.
- You can run a Jupyter Notebook locally or on a remote server, seamlessly working in a graphics- and Markdown-capable REPL in either case
- You can easily share and export the final notebook in its post-run state, presenting Markdown, LaTeX formulas, code, comments, plots, and results in the order they were written and executed, in a single (HTML or PDF) document (or even a slide deck, if you dare)
To your last point about limiting use of the browser to writing short comments and watching videos: I'd say that's a matter of personal taste. Browsers are perfectly capable of quickly handling the volume of code and text that would appear in a reasonable notebook.
That said, I grew up with vim keybindings, and while Jupyter supports the basics (modal editing, j and k to move between cells, dd to delete a cell, etc.) and has super useful default keyboard shortcuts, I still move and type faster in vim. But Jupyter adds so much value as a seamless, graphics-enabled, run-anywhere REPL that I happily accept the slight slowdown in my typing and navigation, which is never the bottleneck in my R&D work anyway.
You can replace "firefox" and "emacs" with other things to take that last argument to an extreme. For example: "My [laptop] isn't customized to type code, it's customized to type HN comments and watch cat videos. Why use [a laptop] instead of [a workstation] to type code?" I thought like that 15 years ago, but technology and I changed.
You use an image-based language (i.e. Lisp, Smalltalk)
https://github.com/LiveAsynchronousVisualizedArchitecture/la...
All the IO between nodes is serialized, so an output that is already in shared memory (to be examined or visualized in other programs) can also be written to disk.
It can then be loaded as a 'constant' node and iterations can be done with hot reloading on the baked output.
They key is that you need both live/hot/automatic reloading of execution AND snapshots/replay/rewind/baking.
If you only have one or the other, you can't isolate a part of a program and iterate on it nearly as quickly.
I have a hard time maintaining discipline in a notebook and therefore don’t find them very useful
It runs code in IPython, but you write in a regular Python file rather than a notebook. I've used it in exactly the manner you describe, doing a load to memory once, then running and rerunning analyses (or parts of them) in the same Ipython console.
Also echoing thaw13579's comment; I used GNU Make in one project [2] for this and it worked OK.
[1] https://github.com/reclosedev/requests-cache [2] https://github.com/kerrickstaley/tatoeba_rank/blob/master/Ma...
But at some point I'm getting to a stage where I realize that I'm starting to write architecture, not algorithms. And at that point the only thing I'll do with my notebooks is glance at them to look at particular solutions, and maybe run them once or twice for sanity checks (or to just be happy that I solved the problem :D).
I then try to flip a mental switch. There are fundamental differences between writing python scripts for scientific or any other purposes and writing python architecture. Which is why I've moved away from copy-pasting code from notebooks to codebases. (I've been stubborn to actually ingrain this for a year or so.). I treat my own notebooks as if they were foreign solutions. Good for knowing how to tackle the problem, but I'll keep them the hell away from my clipboard.
At that point, it shouldn't matter if I have to re-run everything again for tests, because the stability and reliability of my code is now the highest priority. Also, if your development progress is anything like mine, the actual problems you fix a few weeks into a project aren't problems where you try a solution every few minutes, but rather once or twice a day. So the extra time spent waiting doesn't really bother me.
TL;DR: Hack away in Jupyter notebooks for exploration, then lock it all away once you know how to tackle the problem and work 'traditionally'.
How can one get the same type of developer experience in Python without using Jupyter notebooks? What I'm talking about, to people that maybe do traditional development in lets say Java or C++ is, imagine running your code in a debugger, setting a breakpoint, when it hits that breakpoint, you see "ah here's the problem", you fix the code, and have it re-execute all the while not having stopped your program at all. No re-compiling, and then having to re-execute from the beginning. It's like once you've done things the Juypter way, how can you possibly want to back to writing code in a traditional sense, it's just too slow of a process.
How do you get that same experience without using Jupyter? I tried the VSCode plugin [1] that tries to make things the same as a Juptyer Notebook, but it's no where near as smooth an experience and feels clunky.
[1] https://marketplace.visualstudio.com/items?itemName=donjayam...