For machine learning, you want very fast matrix operations for the actual training/evaluation parts and very good data munging for the "get the data from this file/API/database to your ML library". The Python ecosystem is strong at both, which sets up a good feedback loop for even better libraries to be built on top.
That's my take, anyway.
And you could certainly do things like shuffle lots of data easily for more hardcore processing. For example, Symbolics didn't just have a Weitek floating point accelerator option for their systems, they had vector processing boards for them too that you could use from Lisp. (And even a GPU, the Framethrower, that did full-frame HD with 2D and 3D acceleration! S-Graphics is amazing for the time.)
Also, a lot of AI work back then was focused on symbolic processing. That's largely been eclipsed by the more math-heavy approaches these days but symbolic processing is still around, for example in tools like Cyc.
I think Python really took off for that because it already had quality and widely-used libraries for writing the code in Python and doing the work in a more efficient place (numpy, scipy). Clojure has one of those for matrix multiplication but not much else there, and I'm not sure Racket has anything at all.
This is an apples to oranges comparison. The Common Lisp standard hasn't been updated since 1994. The Python standard has not been written at all yet. Lisp and Python implementations both continue to evolve and be released.
https://github.com/mikera/core.matrix
https://data-sorcery.org/
https://github.com/mikera/vectorz-clj
http://neanderthal.uncomplicate.org/
https://github.com/tel/clatrixLisp is incredibly malleable and this may have hurt it over the years.
Also a lot of the latest AI is hyper optimized data crunching on GPUs which isn't necessarily one of lisp's strengths.
I think it is also a matter of culture, probably if the likes of AMD and NVidia cared, they could invest some money into making such languages run properly on GPGPUs, instead of leaving it to researchers alone how to target PTX and ROCm.
On the other hand something like C++17 would already offer many of the Lisp benefits, even if a bit uglier.
Or perhaps my point is, wouldn't it be harder to get old-style (symbolic) AI to run in a GPU than ML-style AI? (If I understand correctly, the old style is a lot of walking data structures, and the new style is largely matrix operations. The latter seems like a much better fit for a GPU than the former.)
Also the declarative way of programming in languages like Lisp, and also the macros, would surely allow for nice expressive DSLs.
So far I am only aware of companies exploring Haskell and F# support for GPUs, but I guess it is usually a matter of someone trying it out.
After all, there is FPGA tooling generation support for Clojure already,e.g. Piplin.
That was very kind.
The main force which explains everything is the procession whereby mainframes were replaced by minis, were replaced by workstations, were replaced by microcomputers.
At each stage, the new wave of hardware started small, bringing in its own approaches, tools and languages. As each wave matured, certain software technologies made "the jump". Some didn't. Some made the jump, but their popularity was destroyed. Those approaches which started each wave had a certain edge, even if they were inferior. For instance, the BASIC language was very widely available on the first 8 bit microcomputers that burst onto the scene in the late 1970's, putting computing into the hands of non-institutional users for the first time. In computing ivory towers of that time, nobody considered BASIC viable at that time any more. Yet, because of this boost that BASIC received, riding the microcomputer wave, it still persists with us in the form of Microsoft's VB.
Lisp was very successful in the 1980's. However, it didn't run very well on microcomputers. If Lisp hackers wanted to make an application to sell on the mass market, the faced the prospect of their customers having to buy expensive workstations and minicomputers. So they did the obvious thing and re-wrote the logic in C or what have you, making it workable on an 8 Mhz PC with a meg of RAM. Once someone has "made it" by doing such a thing, they stop learning. Twenty five years later they are still telling the same story to new recruits about how they rewrote some Lisp thing in C to make it actually run, and made a business out of it, hence forget Lisp.
By the time the next wave of hardware gets to the point that it exceeds the previous generation in capacity and power, it's too late to try to revive most of the stuff that didn't make the jump. The people moved on to something else (or have even become irrationally permanent naysayers), plus other things have obviously changed in the world.
Lisp is doing very well all things considering, because of great expressivity and abstraction, machine independence and overall enduring value. Also, its adaptability: the ability to be reshaped into new dialects. Nothing that old has anywhere near the clout.
As far as the AI winter goes, basically the spiel is that certain funding money dried up for certain types of AI. Even if that is true, what does it tell us? That certain people were dependent on that type of money. They were dependent on it because their stuff only ran on institutional hardware; they were not able to wean themselves off the institutional teat and do something in the mass market. At least, not without changing toolchains.
I have a feeling that would have turned out quite different.
(The reason it's Python and not another popular language is another matter - I think for various reasons Python was a more popular language for scientific computing).
Python is much more popular in general, which makes it more likely to be popular in any specific subfield, for various reasons:
1. More chance that people who decide to do anything in ML already use Python.
2. There's better existing support for various ML-related tasks in Python.
3. There's a larger audience available for ML-related things in Python, therefore people think it's more worthwhile to code things in it.
etc.