I think something hard when it comes to this is defining the resolution or verbosity of what you see. What i mean is that sometimes you want to see implementations etc and sometimes you want to keep something abstracted away and "debug around" it. I feel that the "step in", "step over" controls for this are somewhat too simplistic. It would be nice for example to mark some library functions that i never want to step into. I think debugging is great but there is still a lot of room for improvement.

Yes, that is exactly my feeling. There are a number of dimensions to be covered, like code resolution, the representation of more complex objects (including prior states), visualization etc. Thats before we go into esoteric stuff like C++ bindings.

The "debugging/scripting a data pipeline" task is somewhat orthogonal to building applications or exploring data but these days it is something alot of people are effectively doing.

JetBrains IDEs have a handy "step into my code" control in addition to the usual ones. It's not quite as granular as picking specific libraries not to step into but it requires no configuration and it's often enough what you want.

Actually what would be perfect in my world is when debug adapter protocol would support live reload. You change the line and everything is re-executed / incrementally updated to reflect the change and you are back at your current breakpoint in an instant.

I think this is why engineers like Matlab. It’s not fashionable these days but they get their IDE and docs right.

The feature in Matlab that is great for this is the 'workspace' panel- and it's something I love most about the Matlab IDE.

being able to watch your variables as the program executes, see what was allocated, see what wound up being stored in them, if there were matricies what size they are and being able to stop and open them up and check data, see data types at a glance etc.. really helps people who are just getting started with understanding imperative, procedural code.

There used to be a Python IDE that had the same feature called Rodeo, but it has been abandoned.

There also used to be a Julia IDE called Juno which was excellent at this as well, but it also have been abandoned in favor of the Julia VS Code extension, which has similar features.

Spyder also has a variable explorer. Basically Spyder is often referred to as MATLAB-like IDE for Python and MATLAB is even referenced in the documentation.

To add to this, Thonny has a variable explorer that shows object attributes.

yep, spyder is python for recovering matlab users

Or RStudio like IDE for Python.

The most enjoyable programming experience I've ever had, hands-down, was writing Clojure in LightTable, an IDE that allowed line-by-line inspection/execution, which sounds like what you're talking about.

The project looks like it died unfortunately, but I think they did end up adding Python support before that happened. Might be worth checking out.

I believe this is called the REPL driven development in Clojure, when ever you are writing code you have the REPL available and it’s one of the main selling points of Clojure.

My opinion is that I get very close of this REPL style with a notebook and (right click on notebook tab in Jupyter lab) open console for notebook.

This opens a REPL (iphython console) alongside the notebook. The you can code all kinds of Clojure stylistic Python ways using nested dictionaries.

You have to make sure to copy paste you finished code into the notebook or other .py file not to loose it, but it’s minor.

I've been doing the same thing in VS Code for Julia/Python, but the experience is much worse than LightTable [0]. It's slow, the interface is clunky/janky, and noone has invested effort in designing the UI to make navigation/information clear and accessible.

[0] Admittedly LightTable was pretty buggy so it wasn't all fun and games. But still - I would love for my day-to-day development environment (Dart/C++) to look like that.

That functionality and more is available for the most common editors/IDEs: VSCode Calva, Emacs Cider, Intellij Cursive.

You can also write notebooks with clerk, which gives you a whole bunch of data visualization utility and renders to a browser (via websockets). Also has a static html export. Cool thing here it’s that it uses just normal Clojure files, so all your tooling just works.

That's cool, I knew about Cider but not Cursive/Calva (I haven't used Clojure in years). I would love for this to be more common for other languages (F# in particular would really benefit, IMO).

LightTable was indeed very cool when I tried it.

There is a Haskell IDE available in the MacOS Apple Store than has a similar live view feature. I don’t much use this IDE, but it is cool.

Have you tried debugging with Visual studio code? It sounds a lot like what you're asking for. Stick breakpoints where you want them, you can step through etc, and there's variable viewer in the left hand panel - which isn't great on it's own, but then you can right click on the dataframe and open it in data viewer and it'll open a new tab with the dataframe in a table that you can view, and then if you step further through the program you can hit refresh on that view and it'l show you the updated frame.

Yep, and if you click the three dots next to the debugger config selector on the top left, you can access the debug console and play with a live terminal with access to all of the variables and data that are currently paused.

You know what would be killer, if VS Code debug mode would kick out into a notebook environment.

The repl it provides is pretty close. You can also use special comments to designate cells and just run those parts of a larger program in isolation with a repl, which I have used for debugging

Vscode already has a jupyter notebooks mode with full debugger support if that’s what you mean by this.

I had this idea a few years ago and built an emacs front end for it

Would be really slick in vscode…

https://github.com/ebanner/pynt

Coincidentally, I just tried this yesterday and it was a good experience. No setup time, I just clicked to the left of a line of code to set a breakpoint, and then clicked the debug icon instead of the run icon. The local variable viewer was good enough for me.

One format missing is a canvas. https://natto.dev/ is awesome for experimenting.

Like MATLAB?

This debug-first stepper, complete with workspace view, is now supported in JupyterLab for Python (not necessarily the app) using the xeus-python kernel.

Yes, this is the way to go. I found this was the easiest to use.

I think that JetBrains' PyCharm does an excellent job in the debugging arena.

For the basics, you can effortlessly but breakpoints, click through stackframes, inspect and modify variables via the GUI variable inspector window, execute statements in the console, etc.

A nice extra feature is the fact that as you step in, out and over certain pieces of code, the lines of code that are ran get annotated with the resulting values of the variables that get changed. E.g. when for-looping over an iterator, the value of x gets displayed next to the *for x in y:* line.

I also really love the conditional breakpoint functionality; it allows you to only break out into the debugger when a certain condition (expressed in python by the user) is met. Very handy when iterating over larger pieces of data that have sparse bugs in certain edge cases.

Edit: as a bonus, also quite nice that the vim plugin also works in the debugger console :)

Yep, and it'll display your pandas dataframes and had some notebook support too. <3 jetbrains

AREPL uses a debug first paradigm. It's not step by step, but it does give you access to your variables as you code.

It's free and open source. Let me know what you think: https://github.com/Almenon/AREPL-vscode

The first thing I do after starting a notebook is select 'new console for notebook' which brings up a live console underneath or next to your notebook window as you prefer. The if you hit the little bug icon in the notebook toolbar (on the right, next to the kernel), and the other bug icon in the right sidebar, you get full interactive control and views of everything.

Jetbrains' DataSpell has the nicest notebook UI in my experience - lots of database integrations, R as well as Python, endlessly configurable. It's a relatively new product and still hiccups on some things, eg iPywidget and other interactive notebook tools.

I will drop IPython.embed() break points along an execution path to debug with variable inspection. I tried pdb earlier and attribute to user error on my part not really getting it.

There is also VSCode python hybrid mode.It has variable viewer and debugging.