Most of your points about language deficiencies are valid. But you still have the same problem of longevity and bitrot. Unless you have teams that are willing to pick up Common Lisp and contribute you will end up being the only maintainer of the code. Case in point my advisor from grad school used C and did all his numerical work from scratch. If I'd not found python, numpy and scipy I'd still be in grad school.

If you're looking for a way out of python because of specific performance issues you face I'm not sure why you did not just go with Julia (unless you really wanted to do Common Lisp).

Try pyinstaller if you need to compile python into a self-contained binary.

I agree that code needs to be maintained, but I would say that Common Lisp code has a tendency to bitrot a lot slower than other languages because implementations of Common Lisp don’t break code.

So there’s definitely a valid maintenance point of code needs to continue to be extended (though I’ve had no trouble finding or training Common Lisp programmers), but Common Lisp code simply doesn’t stop working on the timespan of years.

C can bitrot too, especially academic C. A few things I’ve seen are missing headers, builds that only ever worked with what the prof had in /usr/local, conventions that become warnings that become errors (500 invocations of strcpy!), and “beginner” stuff like assuming int’s size or that you can poke hard-coded memory ranges. I do think both languages will be better than Python over decadal time scales.

Excellent comment! I've spent countless hours on trying to get my python environment to just the right state so I can run code associated with scientific papers, and come to the conclusion that it's just a big, unmaintainable mess. There are, of course, many examples of well written python in the scientific world, but there are many times when I simply haven't been able to get it into a running state.

Julia is great in this regard, but it seems that over time, any language with a centralized library repository will become a graveyard for unmaintained legacy code, and this is bad for long term scientific use.

The downsides of rolling your own mathematical functions is that it's very time consuming, and there's always a chance you'll get it mostly right. These are the reasons that python, with numpy, scipy, and the huge quantity of other libraries available tend to draw scientific use.

wrt to old crusty FORTRAN code: scipy is using many of those popular libraries, it is based on them.

wrt to type signatures, many of those ancient FORTRAN libraries are written with implicit interfaces, so bugs are likely to show up. I came to learn this when compared some versions floating around with the patched versions supplied with scipy.

My aim is not to bash, but justify scipy is a solid piece of software, based on known developments, not just a shiny "new" thing.

I don’t mean at all to imply scipy and co are flashy but rickety pieces of software. I think it’s a testament to their quality that such libraries have reached a broad and diverse audience.

I think the foundational libraries of the scientific Python ecosystem are definitely well taken care of. I think a lot of that care comes from “brute forcing” the ecosystem to make it work, eg distributing native compiled C/Fortran code seamlessly on a bunch of platforms with entirely new (at the time) package managers like conda, or wholly new scientific distributions of Python. My observations are more to do what’s built atop them.

> The biggest negative of using Lisp is that a lot of work you’d take for granted in Python or Julia is not done.

Any hard-earned advice on how best to work around this in a scientific context without wasting too much time? That is, which areas of numerics / plotting / stats have a useful Lisp library vs. when it's better to write your own? There are a lot of apparently-abandoned numeric libraries on github. Do you have to learn LAPACK & SIMD/AVX directly to be productive?

For context, I escaped to Python from Matlab 7 years ago, but have grown to share many of your opinions on Python. Looking for where to jump next. The short feedback loop of Lisp (condition system / restarts) is appealing.

I guess my number one piece of advice is to estimate time accordingly. Most things can be solved using pre-existing solutions with a bit of work, if you’re patient and you can afford to put in the time to do it.

Secondary to that:

- Learn to use FFI very well try hard to find libraries written in C.

- Familiarize yourself with the structure of LAPACK and what it offers.

- Learn to use a profiler and debugger (if using Lisp: SB-SPROF, TIME, SLIME, and SLDB).

- (if using Lisp) Contribute useful things back to existing libraries, like MAGICL [0].

In my opinion, Lisp has no good libraries for plotting. I always have to plot by using another tool.

SIMD/AVX are things you use directly in SBCL if you want to achieve very high FLOPS.

Maybe it’s not the best analogy, but scientific programming in Lisp is currently like woodworking (compared to building IKEA with Python).

[0] https://github.com/rigetti/magicl

A list of reasons for using any programming language has to include either "I already knew and liked the language" or "management forced me" to be plausible, otherwise you're just trying to rationalise a choice without admitting which of the big two it was ;)

I do think there’s more nuance to the matter for scientific software since choices are consistent even under perturbations of “what management will accept”. The nuance typically comes from how scientists are first exposed to programming at all, especially late in their career. It’s often a survey course like this so they can catch up and be productive members of their lab. And a course needs to fit in a semester, so any language that’s not Python, R, or MATLAB will be wholly inadequate for that purpose.

More than that even... I spent 4 semesters learning scientific computing in FORTRAN in school. As soon as I graduated, my go to language for prototyping stuff and frankly 99% of projects became and is still python (or c++ for microcontrollers).

The reasons are simple:

1. Lots more help and examples available online for python compared to FORTRAN.

2. Python is performant enough, 99% of the time if you make even a vague attempt to appropriately vectorize your code and/or use the right libraries and techniques.

3. Solving a well defined problem in any language is easy enough. What's hard is getting to that well defined problem, and that often involves playing and tweaking with things until you wrap your head around the problem space. Python saves me a lot of time so I can iterate faster by avoiding stupid errors and having much lower boilerplate overhead where stupid errors can really propagate into hard to follow error messages down the line.

Python just is a lot more intuitive. I don't have to waste nearly as much time on off-by-one and other stupid errors because my indexing was fucked up. So I can spend most my time on thinking about the stuff that really matters rather than implementation details.

That said, I can write some lean and mean FORTRAN if I really need to, and I'm ok with c++ when I need to be too. In reality though, most of my workload isn't that computationally intensive, and even when it is, most of the hard parts have been outsourced to c++ behind the scenes anyway. I can't even remember the last time when my computational need was bad enough that I considered writing my own fortran.

Microcontrollers are a different story. Trying to use python for that seems like a dumb idea, but I know MicroPython is a thing, though I'm skeptical of the whole concept to be honest. You're so close to the metal at that point that why would you want to abstract it away?

There are cases in ML where python isn't performant enough, so much so, that the state of the art for a while converged on how to circumvent Python. This happens when the model interacts with the outside world (environment) a lot or when you are limited to non vectorizable environments. Python simply adds too much friction to the mix.

It may vary from one discipline to the next. My background is in physics. Every physics student learned to program. In grad school, every physics thesis involved programming. Most of the scientists I know who use Python, switched from some other language, and learned it on the fly.

Looking back over the span of 4 decades, one big change is that there is just more stuff, and the specific programming language is just a small part of it. An advantage of learning something like Python is that you can do more with it, quickly, thanks to the libraries.

Well sure, I accept "I use this because that's what the helpful postdoc knows" as spiritually falling under "I already knew it" heading.

I would guess that very few people are reading, editing, or re-running your code, though, which kind of changes the nature of discussing a programming language. Otherwise they'd all need to know and want to use Common Lisp.

This isn't a criticism of your code, it just reflects a larger issue in scientific programming: most code is never read, edited, or re-run by anyone other than one sole author. If that is the status quo, most considerations about languages don't even matter anymore, since the biggest issue is normally just finding a good balance between functionality and ability to collaborate, and scientific culture has lopped off the collaboration side of the equation.

This choice may also feed into that culture, a positive feedback loop of cloistering: if I were in your field and noticed you had a code base, I would have to go out of my way to run your code or improve it, so I probably just wouldn't.

Most of the scientific Common Lisp code I speak of is/was developed on teams at a handful of companies.

Have you used Poetry? Because most problems with managing Python projects and pain points about environment are solved by a single solution – Poetry.

Seriously. Use poetry to manage your venv + deps. It solves 99% of the common problems I find about conda, venv, pip, etc.

What's your Common Lisp and operating system of choice? Are there ones that are easier to do FFI?

Another case of "I like what I know and I know what I like" :-)