Python in science is kinda like the One Ring the Binds them - it is the glue that holds together the HPC code at CERN, ESA, NASA, and more, it crunches climate models and delivers your daily weather forcasts. Some really great projects like Cython, Theano, and Numba boost the performance of numeric code to native speeds with a fraction of the hassle of hand-written C. Mountains of machine learning research is done in Python, as well as statistics and analytics. And of course Numpy remains the de-facto standard of N-dimensional array libraries - if you look at numeric computing libraries in many other languages they will be framed as "Numpy-for-X".
While Numpy/Scipy is great with a capital G, it seems the underlying infrastructure is flaky.
I'm talking about the messy process of building and installing it. And it seems everything is held with duct-tape
Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed.
Most any significant scientific package I've ever had to build (which is quite a few) had a complex build system that felt like it was built on top of shaky glass pillars. It's just the nature of what's involved and it's a fundamentally hard problem; it's difficult to replace those libraries because the extreme amount of verification/validation and optimization that have gone into something like LAPACK makes it economically undesirable.
I'd say also that this is part of the appeal of NumPy and SciPy, which is they glue together a huge number of tools that are otherwise disjoint. You don't have to worry (most of the time) about matrix ordering, calling conventions, those mysterious "workspace" variables in LAPACK etc.
That and Cristoph Gohlke[0]'s repository of pre-compiled modules. That guy saved me more than once. Should I ever meet him, I'll sure buy him a beer!
If you're using Python for scientific computing, you can use the Anaconda distribution of Python, it makes it painless to work with those packages.
> "Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed."
Python has excellent ORMs, I'm not sure what you're getting at.
% pip install numpy
For more (e.g., other OS'es, build-from-source), see:
My C code was very complicated in order to be fast. Lots of pointer swapping, bit-shifting, etc.
The code was about 4% faster in C, which was more lines of code, much harder to read (and write), compiled with every imaginable flag that could offer a speedup. Switching operating systems, this code was about 8% faster in C on Linux than Python in Windows 10. C in Linux would take about 9.5 days to complete, Python in Windows 10 would be about 10.5 days.
In addition, Python has the "pickle" module so that I can easily make the algorithm save state to pause and restore it any time I want. What's more, I can probably use mmap to share memory easily without making copies and switch to multiprocessing for an even greater speedup (my first attempt using Pool was a little slower despite using 3 cores, but that's the slowest way to multithread).
Plus, with each variable being ~10Mb, I was concerned about memory copying (I studied mechanical engineering, I missed a lot of this stuff in school), and with C I could make memory copying vs passing pointers explicit. All of that was unnecessary.
It was another reinforcement of the idea: "Build it first, then optimize".
Picking the best tool for the job often gets in the way of actually getting things done. Python is the true swiss army knife of today's programming languages.
Not advocating for Perl, just on behalf of Perl lovers like my office mate
EDIT Added another "most" to make it clear that not everyone can write readable python.
What it offers for me truly is the batteries included and easy, memorable and predictable syntax. (Probably due to experience).
Sure, Go is easier to deploy, Erlang and by extension Elixir are easier to scale.
But python does align very well with my personal though process. I don't think parallel and I don't have race conditions in my mind. If I want to quickly write something that will perform decently, I choose python, because the time I save by not having to learn a new language could be days or perhaps even weeks. I guess it boils down to this: with my current experience, I'm more then 10 times as productive in python as I would be in any other language.
Sure, it would be neat if I had as much experience in Erlang as I have in python, but that will take years, and at the end of the day, I'd rather finish my project this year with python, then next year with any other language.
Python finally has module packaging down fairly well (Go still drives me crazy even with the latest vending approach--cannot wait for a major build to break because the module creator switched hosting services).
Python is pretty pleasant to debug. The ability to set breakpoints and then poke around and/or modify values saves me hundreds of hours a year.
Building executables can still be a bit problematic (one reason we used Go for a few projects -- they really have this down). However, latest pyInstaller has been working smoothly on Windows and Linux.
Python is quick enough -- and we know how to scale websites with WSGI and stand-alone apps with asyncio (or Twisted). Rarely does the need for that last ounce of speed outweigh the ability to quickly create and modify apps.
Elixir, of all the new languages, intrigues me the most; not just for the language but being built on top of Erlang's OTP platform. They also seem to be doing a decent job on the packaging and distribution front.
TBH, that's the most rational reason ever for choosing any language. Anything else is much less important.
But, yes, for me the experience is the opposite, Python flies in the face of my thought process in every single aspect of it. I find it limiting, inflexible and dominating. Which means, everyone think differently, and there is no single language that suits all possible combinations of problem domains and thinking styles.
Go's lack of a package manager making deployment more painful is the thing that made me come back to python.
(Does go statically link the libc, too? If so, you should be rebuilding all your go applications right now, after the glibc issues reported on Tuesday…)
And it does the job.
Mastering Python-foo is still on my todo list, though. As a Ruby user, I've noticed that Python scripts are much more straight forward and maintable. I still write my scripts in Ruby though because nobody seems to mind. This is perhaps because I try my best write good comments. In general, Ruby is better as Perl replacement than a Python substitute. In fact, one of my coworkers who is an oldschool perl scripters was sold on Ruby over Perl just by reviewing one of my scripts. I think the issue is people don't think of Ruby outside of Rails, where it's basically a readable purely object oriented Perl.
As an aside, I think there is a bigger overall problem of using scripting languages for application development. I wouldn't use Python, Ruby, or Perl for anything that was more than, say, 500 lines. I almost cried when I tried to read the code for homebrew. Not that it's bad code (it's quite good actually), it's just way too much duck typing for me. I could never keep all that type information in my head. I prefer to pass that cognitive load to the compiler.
https://migrateup.com/main-difference-python-3/
Perhaps if python3 annoys you more than makes you say, "Hey, this prevents lots of latent bugs I might have easily shipped to production before, cool!", then you need to learn python a bit better and re-evaluate what is annoying about it.
What I was trying to say is that the portability of python3 is bad enough that it effects non-python programmers.
For example, sometimes I want to try a new vim plugin that's written in python. I install it, and it doesn't work. Great. Then I find out it's because OS X's python can't interpret it. So I install python3 (which has a different executable name, also annoying). All good, right? Wrong. Vim needs to be compiled with python3 support because the python extension is exclusive to python. Now I have to recompile macvim with
brew install macvim --with-python3 && brew install python3
It's not a quick process at all. Especially when I'm just trying something out. This stuff happens with other tools I've wanted to use too.It doesn't ruin my whole day or anything, but it's a bit irritating to have to manage several versions of an interpreter for a language I'm not even using that language in production. It doesn't help that I use several machines and VMs.
So yeah, I know exactly what is annoying about it.
If you like Go, try Nim - it's like Go but without all the dogma. I wrote a blog post about my little Nim project: https://klibert.pl/output/slock_in_nim.html
> Why should I choose Python 3 for my new projects?
You shouldn't unless it fits all the constraints of your new project. Choosing a language is not, for the most part, a technical decision: sure there are language features and implementation features (those are separate matters!), but in the end it comes down to external factors: your experience with a particular paradigm, your taste in syntax, skills of potential collaborators and so on.
One thing of note, though: there is no rule saying that you can't use more than one language for your project. On the contrary, you can mix-and-match languages with relative ease nowadays (see https://klibert.pl/output/python_interop.html) and, in my experience, it works rather well. For example, in one of my projects I need to scrape a bunch of websites for data and then process this data. I wrote scraper in Elixir, which made dealing with network errors and parallelizing requests a breeze, and used Python with Pandas for the analysis part. It was a very pleasant experience: neither Python nor Elixir were up to the task on their own, but they worked very well together. It's even easier when you use a JVM, .NET or JavaScript-based languages.
In the transpilers era, Javascript will never be so bad that choosing another dynamic language is a clear win. I'd go back to Python and Django for a CRUD app, but since Javascript is easier to write in a functional style than Python, I've actually started to prefer it as a first choice.
For companies, there are more factors to consider, eg. finding devs at a correct price, getting existing workforce to work on a new tech, size of standard/open source library, getting help and bugfixes from community instead of hacking on your own, risk of language/tech losing steam, etc. That's why several comparatively "boring" languages still thrive.
As for Py3, the standard libs are very clean and streamlined, (e.g. io). Oddly enough, the point that I love about Py3 is its handling of unicode vs bytes (which most people find it difficult for some reason).
I'm not sure what you mean by "can't target the browser"? I don't find distributing/deploying python to be difficult or even involved at all. I do get annoyed at the single CPU bound limitation.
If I had to summarize why I'd pick Py3 for new projects, I'd say because it thrives in being minimalist and I hate clutter.
I posted about it:
What language has decorators, generators, comprehensions, modules, but isn't Python?
http://fourlightyears.blogspot.com.au/2015/08/what-language-...
Well, yeah, I'm sure ES2105 is cool, but most of us don't have the choice of using languages from the next century.
> What language has decorators, generators, comprehensions, modules, but isn't Python?
Clojure, Haskell. :-)And the worst thing is, Node/v8's JIT would kill Python in performance for a;; the kind of things people write Python scripts for, even if it run all synchronous.
It would be nice if callbacks/promises/async-await were only optional and for network/web things, not for everything.
I suppose you should first figure out the domain, then figure out the language support and ecosystem around that domain and then pick one. That is, if you are implementing something practical whose aim is to provide value outside of personal learning.
For example, several constraints still lead to a situation where C++ is a prime choice unless one is mostly interestes in hacking in a new domain and not finishing a project.
Note! I'm not dissing trying out languages for the joy of it!
Personally, I would pick Python for experimenting with syntax transforms, data mining and such. And as a basis for any offline tool that needs to chug bits from one place to another. Also, the numerical use cases listed elsewhere.
Basically, for anything that does not need a GUI.
The python users I've encountered fall generally into two camps:
1) The engineer/scientist/researcher/hacker type that has python installed on their snowflake machine, and has a whole bunch of packages installed to make their flavor of snowcones. These users just want python to work, find joy in the syntax, and in general get their work done.
2) the developer that is trying to deploy the same thing to many machines. This is a pain. Often you end up with users from camp 1 trying to do this and you end up with a global python install with batteries and the kitchen sink included that is a dependency for a whole bunch of "server" apps. you update a package (because "my special snowcone needs it") and you break something else that you didn't know to test. This is a nightmare. With pip and venv it is easier to manage this, but I still see that relocatable venv is "experimental" when you read the help. IMHO, the python 3 community needs to make it very, very easy to distribute and deploy my special snowcone maker and all its dependencies to a system without python already installed, and not stomp on anybody elses snowcone maker in the process.
I spent a few days going back and forth - comparing the performance, design time, etc of both - and in the end I'm just using Qt's JS JIT for dynamic list comprehensions and other things C++ is bad at. In the end, the fact that distributing on everything but Linux would have been reduced to having to bundle CPython in was just a deal breaker.
Linux isn't really bad for deployment, though. You can use the SUSE OBS to build packages for all the major platforms besides Arch, and since my main desktop is Arch, I'm building my software with PKGBUILD's from the get-go.
Fully cross-platform.. easy to distribute. Quite nice all around.
sudo pip install [package/repo]We package virtualenvs using fpm, others use Docker. But Go just spits out a single binary you can scp anywhere (the build process OTOH is more painful IMHO).
That’s why I’m highly curious about https://us.pycon.org/2016/schedule/presentation/2122/ .
Productivity, productivity, productivity.
Yes, there are faster, safer, and more modern languages out there. However Python for me still hits nails the hell out of getting stuff done, really really quickly. That applies both to the language itself, and from the standpoint of having such an extensive and mature ecosystem of packages to choose from.
My issue is I always see Python as the Second Best choice for what I am doing. There is always a better tool for the job. The one thing going for Python is that it is such a generalist BUT when doing work I prefer the Right Tool for the Job approach. I seriously can't think of a job that Python does best.
For me personally it is the best choice for non-throwaway shell scripting once the scripts becomes more complex and/or long lived than a sequence of shell commands.
Python is part of the base install of just about every mainstream Linux distro, so in that regard its only real competitor is Perl.
So the question for me, for new projects is: why choose Python 3 (over all the other non python options) at all? Not what's better about Python 3 than Python 2, but what's better about Python 3 than those other choices. These other choices all have certain areas where they have great strengths. What is Python 3's strength when compared to these other languages? It's a mutable, imperative, single CPU bound language that can't target the browser and is extremely difficult to distribute and deploy. Or have these issues been fixed?
I don't mean to troll any Python fanboys. I use Python 2 every single day at work and I'm very happy with it. But what does Python 3 offer that other new languages don't? Because when you are starting a new project you can choose anything. Why should I choose Python 3 for my new projects?