Norvig's PAIP is all about manipulating symbols. It's so fundamental that in any other language you'd have to Greenspun an interpreter first, just to solve the same problems in the same way.
Today, AI seems to mostly mean matrices. If that's all you need, then you can do it in any language, especially if you're just wrapping old C or Fortran libraries.
I haven't done any new-style AI but I get the feeling that most "research" is in finding new nails to pound with the same hammer. That's a legitimate area of research, but it also should be no surprise that the concept of "AI research" expanded so much in recent years because it changed from "learn to build your own lightsaber" to "pick up a hammer and hit something with it".
Most of the time, people does not leave Python when it is not satisfactory. They stick with it because they have no clue, most of the time.
For example, it is not quite natural for Python folks to even think about creating rewrite rules (as in Haskell) for programs to stay at higher level of expression. E.g., if you use dictionary with default value, why not lift all operations up (inline, so to say) and avoid calling these that are overloaded? It bit me many times, yet other users of Python just gave me blank stare when I tried to explain these things.
Haskell has two very nice libraries, bytestring and vector which use rewrite rules to have performance comparable to C code.
Operations on dictionaries with default values implemented using inheritance from dictionary. This means that instead of just having conditional expression on the right side I have a call to virtual function and there is an expression on the right side of assignment or return statement.
Most of OO virtual machines can JIT things like these into efficient machine code, by specializing. The same is also quite possible with bytecode like in CPython - by optimization in the compiler or by introduction of rewrite rules like above for library implementors to specify rules that apply to any (transformed) user code.
I must admit I don't quite understand what rewrite rules actually are; are they akin to macros?
Rewrite rules fire when compiler encounters a chance for them to fire, at any point of (optimizing) program transformation.
Due to this fact, rewrite rules are not local - compiler can transform the inner structure of a program changing the locality of program statements.
bs = map g xs
... many other lines of program text not reassigning bs
as = map f bs
Yes, there is a lot of multiplying matrices. But you need to think of a matrix as like a <div> or something and multiplying matrices and placing two <div>s next to each other. Then, think of how to create a dynamic website from that. There is a lot of architecturing involved other than multiplying matrices.
90% of all programmers are cargo-culters whose knowledge of programming is gained from the answers they cut-and-paste from Stack Exchange or Quora. This is as true of "data scientisis" as it is of "web developers." The result is that there is a network effect making certain answers determinant of the language or monthly flavour of xxxScript that gets pasted most often, and reduces the domain in which further answers are examined.
If you want it in simpler terms: the new dudes looked around and saw all the cool kids were using Python and only a few boffins (UK) or nerds (USA) were using anything else, so they knew who to crib the answers from if they also wanted to look valley cool.
Its ubiquituous in academia/science, has a low barrier to entry, and a large amount and arguably often unmatched quality of specialized libraries/frameworks/tools for all kinds of scientific fields. So the first go to language/ecosystem will of course almost always be python.
I'm definitely not the biggest fan of Python (the language). And there certainly are more elegant/fitting languages for many problem classes. But the thing is, from my experience many people in the field are quite aware of this. But it simply doesn't matter if it's the best language from a developer POV. It just has to be good enough to get the job done, apart from this the points above are simply more imporant.
And it's hard to understate the importance of ubiquity in this case:
Python is the code equivalent of a lingua franca in science.
Scientists are smart within their fields, but they're usually terrible at programming. I once saw a hundred-page source file (one file for everything) that defined no functions. And of course it was in python.
So if you're arguing that python is good if you want to exchange source code with people... I guess you might do that? Personally, nobody has asked me for my code in quite a while, so I do all my work in Calysto scheme (which is written in python and can accesss the whole library ecosystem).
If, on the other hand, you're saying scientists' opinions on what constitutes a good programming language ought to be taken seriously, then you're wrong. Scientists are crappy programmers for the same reason they're crappy writers. Programming and writing are tools they use to get their real work done. Most of them master the minimum they need and spend most of their time in the lab
I read open source, then extract the bits of IP I think was useful. Often rewriting it to fit my case. Its useful source, but as an example to help me write my app.
{ Please don't use Linux as a strawman (You rewrote Linux?! I don't believe it!) That's the single largest, most successful (along with maybe 5 others) open source project. But absolutely not representative of the 1M other projects out there. }
Different communities have different needs. Data scientists, or physicists, or biologists don't care about memory safety, the programming language is a tool to run a computation for a limited time (not a server). The tool should not get in the way of fast iteration and rapid prototyping which many languages do in varying amount: C with memory management, Haskell with category theory, Rust with borrow checking, C++ with being C++, ...
Basically you are criticizing data scientists for choosing to not invent a new thing every time and reusing an established ecosystem.
I agree that people typically want to get from start to solving their problem as quickly as possible, and don't care about things that don't obviously contribute to that. But that doesn't mean they are perfect rational agents, or immune to cargo-culting, or that a little time invested in learning something else wouldn't pay off.
Here is my data science stack https://github.com/mratsim/arch-Data-Science
Let's go over it:
- Is there a mature tensor/multidimensional array library in Clojure? Is there a Dataframe library?
- Are they compatible with common formats: .npy, .csv, .hdf5 and sqlite databases
- Is there a fast PCA and DBSCAN algorithm?
- Can you pass those arrays to XGBoost or LightGBM for Gradient Boosting based classification or regression.
- Do they support out-of-core processing (i.e. memory-mapped arrays) for processing that takes more than your RAM (happens often with geolocalisation)
- Can you use PyTorch through Clojure? Can you use Tensorflow? What's the GPU computing story
- Does your ML library support things like One-Hot-Encoding, cross-validation? Or more advanced like Target Encoding?
- NLP: stanfordNLP wrapper? spacy? Huggingfaces? Rasa-nlu? sentencepieces?
- Deep learning: Do you support image augmentation like Nvidia DALi or Albumentations? Mixed precision training like Nvidia APEx
- Vision: How is the OpenCV support? Does it interface with Clojure multidimensional arrays? Do you have morphological operations like erosion or dilation?
- Visualization: is there a dataviz library like plotly, ggplot2 or Altair/Vega?
And more importantly
- Are there tutorials and courses in Clojure from beginner to advanced concept? Even Julia is struggling with that.
And note the question you didn't ask: does Java do anything better than python? Didn't even cross your mind.
Clojure is certainly the context I'd use Java's machine learning libraries in.
> - Is there a mature tensor/multidimensional array library in Clojure? Is there a Dataframe library?
Is there one in Java? If so, what?
And so on with his other items. It doesn't have to be exactly the same (offering him the benefit of the doubt), but it has to offer substantially the same functionality with substantially the same performance. So, you want to use Java libraries from Clojure? Go for it. Where are the things he needs?
https://github.com/originrose/cortex
https://neanderthal.uncomplicate.org/
https://clojurecuda.uncomplicate.org/
https://clojurecl.uncomplicate.org/
https://github.com/uncomplicate/bayadera
https://devblogs.nvidia.com/grcuda-a-polyglot-language-bindi...
Also since Pytorch 1.4 there is initial support for Java, built on top of the new pure C++ API.
Science is about functions, not objects, a language that requires all functions to be associated with an object is a non-starter.
Earlier on, software engineering wasn't as glamorous as it is today, from either a material or cultural standpoint. I even recall a staggering difference now versus twenty years ago, when I first got into it, which even then was right before the bust of the first dot com bubble. While ostensibly by that point, the field of software engineering as a creative pursuit and an intellectual space had already thoroughly lost its pioneer spirit to the metaphorical urban and suburban "gentrifiers", there's no denying that the field itself continued to grow, and even at that point is a small fraction of what it is today. If software has truly eaten the world, isn't it possible that the languages and ecosystem that survived and reached dominance outcompeted the other ones?
That is to say, what makes you believe that another language is more fit to be in the position where Python is? While I will use whatever language is necessitated by the job (and have never had issues doing that) and even prefer the aesthetics and "handling" of Ruby and modern JS, Python is still the tool I reach for most often when I need to get some general purpose scripting or systems design task done. The ease with which I can go from zero to prototype and prototype to production and production to pain-free maintenance is unparalleled. The only two ecosystems that are remotely close are JavaScript and Java, and ostensibly Rails for web dev. Nothing else is competitive.
What would make you dispute this?
Is Haskell a bad language?
No, it's the people getting actual work done that are wrong.
In ML as in PDE, correctness ultimately springs from performance, be it processing more data or using a finer mesh. So at the end of the day impedance matching with imperative linear algebra is pretty much a must.
The real answer is probably inertia, but ...
I think ultimately there isn't a ton of actual "programming" happening in ML, more implementation. People like Python. It's simple, it's clean, it's elegant and it's performant when it needs to be. I'm sure a lot of people who have thought "man why don't I just re-write this in [language]?" stopped short because they soon asked themselves "wait, why?" Most of your work is on either side of the algorithm (data - [ml] - analysis).
In short: you are better off writing a compiler toward the GPU and use it as a compilation target. And a lot of things argues that language benefits from being a functional one, given how GPUs tend to work on data.
If there were no prior work, and if we have the full GPU specs, I'd reckon you want languages which are strong in program translation: Haskell or OCaml comes to mind.
Elixir---as well as its Erlang sibling---are really good at handling problems where there is concurrency involved. It is subtly different from parallel computation in that you often have many threads of execution all wanting to do different things at the same time. It is the same space e.g., Go occupies really well.
And there are still plenty of cases where this will leave performance improvements of an order of magnitude on the table, because the compiler isn't smart enough to work out the optimal sequencing of operations. Often the computational core which benefits from running on a GPU is small and its complexity low enough that, if performance is a priority, you are better off writing it in an imperative low-level language as a library which can be accessed from higher level code.
I'm asking because I'm part of the team that's working on the Futhark language[0], which is exactly what you're describing: a high-level array language in a functional style, targeting the GPU. We're always on the lookout for interesting applications or benchmarks. In particular, we're interested in knowing exactly where our shortcomings are, compared to hand-written GPU code, so that we may work on improving them. For the curious, we think we're doing reasonably well compared to benchmark suites such as Rodinia[1], though there are still cases where we cannot keep up.
[0]: https://futhark-lang.org/ [1]: https://futhark-lang.org/performance.html
That's only true if your calculations are mostly the kind of things that take advantage of GPUs, like working with deep neural nets or graphics. For most other things using multiple CPU cores will be better.
Because very often the best solution does not win, which is why we are still using x86(_64), Unix, etc.
Python is a good example of worse is better. A functional language could provide more elegant abstractions. But for most people the ML APIs that Python provides are good enough. By taking a pragmatic approach, Python was faster to capture some niches, including machine learning and data science. Now we are at a point that even if another language had better abstractions and had similar library support, it would face an uphill battle because of inertia. Why would someone invest in learning another language for machine learning when Python is good enough?
To me it seems that only a language that has both a good model development story and a good deployment story stands a chance to compete with Python, since deployment (especially on mobile/embedded devices) is a weak point of the Python ML ecosystem.
Attention to language interop (especially with C) and a decision not to try and be all things to all programmers was another reason for its success - particularly in data science which relies upon heavily optimized numerical processing libraries that are mostly best written in C, C++ or Fortran.
The reason why Unix and Python win is simplicity. For many people simple == better.
Software is so easy to create and mold that many get trapped in excess complexity. Not being able to write complex software on a platform such as Unix or Python is therefore a feature, not a bug. It prevents you from ever getting trapped in the wet sands of complexity.
If there is a silver bullet in software engineering it is simplicity - and aggressive pursuit thereof.
Except that Python is not that. Multiple inheritance: check. Operator overloading: check. Functional programming: at least some of that (for-comprehensions, map, filter, reduce provided by the language). Exceptions: you got it. Coroutines: sure.
Python provides all the rope to 'get trapped in the wet sands of complexity'. Add the lack of static typing and you can write impermeable systems. There are a lot of Python code bases out which are exactly that. Go is a simple language, Python is not.
Worse is better" is a myth
Richard P. Gabriel:
The lesson to be learned from this is that it is often undesirable to go for the right thing first. It is better to get half of the right thing available so that it spreads like a virus. Once people are hooked on it, take the time to improve it to 90% of the right thing.
Or as aptly summarized by Wikipedia:
Therefore, the worse-is-better software first will gain acceptance, second will condition its users to expect less, and third will be improved to a point that is almost the right thing.
Which is exactly the history of Python (and Unix), compared to 'better' languages of the same era. Note that 'worse' is not necessarily used in a derogatory way. It just means that it is often better to get a good-enough, working, simple implementation out there that quickly adapts and spreads, than making that unicorn perfect system. If you look at the early Python versions and how Python spread over the years, it's a prototypical 'worse is better' example.
Go and other languages listed in the original articles on the other hand require you to understand the full complexity before you even begin or spend most of your time deep in documentation if you decide to learn as you go. Static type language simply doesn't allow you to not understand the type system.
This effect is especially enhanced in Scala, Haskell and similar ones where you will not finish a thing until you get a degree in category theory or Odersky theory.
'Worse' might not be intended to sound bad, but it surely does. It seems it is used to describe Unix and Python by that cohort of programmers whose intention is not to get shit done but to develop some mathematical theory of everything.
People who care about delivering results would not call them 'worse' compared to current alternatives in the first place.
There definitely are better alternatives on the horizon and they are better because they bring more simplicity to the table but not at the cost of giving up intuitiveness.
This is simply not true.
>Which is exactly the history of Python (and Unix)
I was thinking Python (and Windows) :v
Isn't that the whole point of worse is better?
To model complex systems you have to keep them as simple as possible.
Imo the answer is that only one mainstream language bothered to emulate MATLAB's plotting API to the best of its ability.
Bitsavers has all the necessary papers and manuals, and computer museum as well.
Here are some video sessions.
"Yesterday's Computer of Tomorrow: The Xerox Alto │Smalltalk-76 Demo"
https://www.youtube.com/watch?v=NqKyHEJe9_w
"Alto System Project: Dan Ingalls demonstrates Smalltalk"
https://www.youtube.com/watch?v=uknEhXyZgsg
"Eric Bier Demonstrates Cedar"
https://www.youtube.com/watch?v=z_dt7NG38V4
"The Final Demonstration of the Xerox 'Star' computer"
https://www.youtube.com/watch?v=_OwG_rQ_Hqw
The end of the road from Plan 9, Inferno.
Of course, out of these, NT did succeed wildly as well did L4 (e.g. iOS secure enclave).
Some improve in the architecture, some in user land, some in both :).
Many people make the mistake of promoting the language instead of the framework.
And the next thing is support. When some big company releases a ML framework and starts using it for themselves people start using it for their own projects. It's mostly because people think that it will last longer then a month. There were lots of dead python ML projects before Tensorflow.
So if you can get a big company to release a ML framework for your favourite language it will surely gain at least some market share.
The language almost doesn't matter. I say almost because big companies are more likely to choose popular languages.
Nobody else seems to have tackled this specific issue so I'd like to.
Matt's use of "expressive" is not whimsical or full of personal ideology. He uses Felleisen's definition of "expressiveness" when comparing programming languages [0][1]. The work is a continuation of Landin's earlier efforts to formalize language comparison. From the text (where Felleisen is comparing two languages of "different size", which means that one provides more first-class "constructs" than the other):
> Here "more expressive" means that the translation of a program with occurrences of one of the constructs c_i to the smaller language requires a global reorganization of the entire program.
The paper goes on to provide a mathematical framework by which to make these comparisons.
The languages Matt lists are more expressive than many non-functional languages for various reasons. Each of them has some functionality that, if you were to remove it, would require a "global reorganization of the entire program" to achieve the same goal. For example: if you have a Haskell program and remove automatic currying, you may now have to rewrite almost all of your functions and call sites to accommodate (perhaps by explicitly encoding delays, or by using continuations, etc.). Python does not feature automatic currying, so in this respect Haskell is more expressive than Python — there is a method in Haskell in which you can express your code which you cannot use in Python.
Note that the relationship is not predicated entirely upon individual constructs: it is possible for two languages to be compared and to find that each has expressive capability that the other lacks. However, by and large the functional languages Matt lists have focused specifically on increases in expressive power compared to the more "traditional" languages that most programmers use daily. (Haskell did this explicitly by design, but the others Matt lists came to it by way of exploring alternate approaches to problems and arrived at increased expressiveness incidentally.)
Later in the paper, Felleisen touches on Turing completeness (a common point brought up in these discussions):
> If a programming language can represent all computable functions (on the integers), it contains a functionally equivalent counterpart to each program in a more expressive language. This raises the question as to what advantages there are to programming in the more expressive language when equivalent programs in the simpler language already exist. [...] > > [...W]e have come to the conclusion that programs in less expressive languages exhibit repeated occurrences of programming patterns, and that this pattern-oriented style is detrimental to the programming process.
So this provides motivation for why one might want to investigate these "more expressive" languages. Being able to reduce code duplication is (almost) always a good thing!
---
[0] https://doi.org/10.1016/0167-6423(91)90036-W
[1] I know this to be the case because I've talked to Matt about this very article. :) The Felleisen paper is pretty popular in the PL communities Matt used to hang out in.
Whatever language you end up using, the low-level implementation must allow you to precisely control memory layout.
This is what is being done with C++ + Python but pure Elixir doesn't allow you to control memory layout and will kill your performance.
Why bother with Erlang as a thin wrapper over C++ when C++ has the necessary facilities already?
Also someone seems to be doing interesting work to integrate with GPU at https://github.com/sasagawa888/deeppipe2
Of course that's not to say that Elixir is ready to replace Python tomorrow. I just wanted to mention interesting projects in this area.
[0] https://docs.rs/rustler/0.22.0-rc.0/rustler/resource/struct....
mostly the same reason new developers dont use functional languages for anything else either.
This is definitely a dated article, since the obvious choice today would be Python, but I have to wonder: why aren't the "more expressive" (which in this post seems to be a synonym for "functional") languages used in these fields? I don't have much experience with ML languages, but it seems to me that something like Elixir (where the whole language is centered around pipes, parallelization, and operating on data) would be ideal for data science with a performant native library to back it up.