News for hackers. Stuff that matters.
Re:Python, Machine Learning, and Language Wars (2015) by gtrubetskoy on Tuesday March 22, 2016 @01:09PM (#11338027)
I'm surprised the article doesn't mention Anaconda, which is Python with all the things he lists pre-installed for you. I've been a fan for some time now: https://www.continuum.io/why-anaconda
Re:Python, Machine Learning, and Language Wars (2015) by smortaz on Tuesday March 22, 2016 @03:59PM (#11339532)
Another data point on Anaconda from microsoft's Azure ML team - we've been using Anaconda for the past few years and have been very happy with it. One key feature we like is that we can deploy a nearly identical suite of pkgs on Windows, Linux and MacOS w/o having to worry about mismatched versions, installation/update issues, etc.

Case in point - the Azure ML Jupyter notebooks run on a farm of Linux/docker machines:

https://blogs.technet.microsoft.com/machinelearning/2016/02/...

Once a model is built, it can be deployed for production/scale to the Azure ML backend which runs entirely on a farm of Windows machines.

Both environments have matching Anaconda distros running underneath which makes running on two different OS's practically a non-issue. In fact 95% of our users probably have no idea their notebooks & production site run two OS's (in large part thanks to Python, Linux, Docker, Anaconda, ... and lots of other great open source software).

Re:Python, Machine Learning, and Language Wars (2015) by ktamura on Tuesday March 22, 2016 @05:21PM (#11340041)
I totally echo this. In my opinion, if Python was the Linux Kernel, Anaconda would be Ubuntu: i.e., it ensures that you are actually up and running within minutes and not struggling in the missing-header-files or dependencies-management hell.

Of course, there's time and place for eschewing Anaconda, especially if you are trying to ensure your installation and dependencies are minimal. That said, for most data scientists who are workaday *NIX users at best, Anaconda is such a productivity boost.

EDIT: Actually, my analogy needs further qualifications. You _can_ get into dependencies/missing-header-files hell in Ubuntu, but it usually doesn't happen in the first few hour/days. Both Ubuntu and Anaconda have great "first 5 minutes to 5 days" experience, and that goes a long way in ensuring the adoption of underlying technology.

Re:Python, Machine Learning, and Language Wars (2015) by alceufc on Tuesday March 22, 2016 @01:23PM (#11338159)
I also think that Anaconda is great. However, I hope that in the future we could install numpy, matplotlib, jupyter, etc. just using pip.
Re:Python, Machine Learning, and Language Wars (2015) by zo1 on Tuesday March 22, 2016 @01:34PM (#11338266)
I did that earlier today:

pip install jupyter

pip install numpy

pip install scipy

pip install scikit-learn

pip install matplotlib

The only problem I had was with OpenCV, which requires manual make installation if you want the contrib package. The other problem was when trying to install scikit-learn, it requires manual pip installation of scipy.

Re:Python, Machine Learning, and Language Wars (2015) by timClicks on Tuesday March 22, 2016 @04:03PM (#11339563)
I would always be worried with installing numpy/scipy via pip that I wouldn't be linking to BLAS/LAPACK correctly.
Re:Python, Machine Learning, and Language Wars (2015) by rhodysurf on Tuesday March 22, 2016 @04:17PM (#11339665)
What do you mean? I have always installed via pip with zero errors on all three platforms.
Re:Python, Machine Learning, and Language Wars (2015) by moyix on Tuesday March 22, 2016 @07:04PM (#11340658)
The worry is not that it will throw an error, it's that it will silently link against lower-performance math libraries and make your code inexplicably slow.
Re:Python, Machine Learning, and Language Wars (2015) by yeukhon on Tuesday March 22, 2016 @02:59PM (#11339027)
The reason you can't is because of the C libraries levaaged must be installed prior to install numpy and scipy.

For example, you can't get through PyYaml unless python-dev is installed on Ubuntu. I am not sure if wheel would fix it but I don't think so.

Re:Python, Machine Learning, and Language Wars (2015) by takeda on Tuesday March 22, 2016 @03:43PM (#11339401)
Wheel is a binary distribution, so files would already be compiled and therefore python-dev would not be needed anymore.
Re:Python, Machine Learning, and Language Wars (2015) by yeukhon on Tuesday March 22, 2016 @03:45PM (#11339421)
Is wheel not default? Or play it elementary :-) how would I avoid this issue in the future?
Re:Python, Machine Learning, and Language Wars (2015) by takeda on Tuesday March 22, 2016 @07:34PM (#11340833)
I did not write anything worthy to send to PIP so don't know exactly but looks like it is up to the developers[1]

Anyway I just noticed that PyPI only supports binary packages for Windows and Mac OS X. Although, you could still generate wheels of packages that you use by using something like this:

    pip wheel -r requirements.txt
You can then install them with pip install <file> or (unfortunately I forgot the option, perhaps it was -i) you can use an option to point to a directory containing wheels and pip install to install the main package. It should use all dependencies in that directory as well.

[1] http://pythonwheels.com/

Re:Python, Machine Learning, and Language Wars (2015) by techdragon on Wednesday March 23, 2016 @05:05AM (#11342990)
the issue with Wheels on Linux, and this is very much a Linux problem, is that the 'pre built' nature of libraries in wheels doesnt play nice with the raw chaos of Linux's package management + distro + kernel ecosystem. I raised the question of FreeBSD and Solaris based wheels at a PyCon when I was in a face to face discussion with someone more knowledgeable, and the answer was 'in theory that should work like Windows & OSX, no one has done the hard work yet.'

So yeah Linux is not the most friendly environment for Python Wheels.

Re:Python, Machine Learning, and Language Wars (2015) by p4wnc6 on Tuesday March 22, 2016 @01:49PM (#11338399)
conda supports pip installation within the context of an environment (much more gracefully, in fact, than does virtualenv). So you're not giving anything up by using conda in this regard.

I don't see the desire to have pip as the baseline. For me, the conda packaging is much more informative and placing everything you need for multiplatform support into an /info directory with a meta.yaml is a lot more effective than going through the steps of PyPI. conda also makes uploading and hosting on anaconda.org extremely easy.

Normally there is the whole "gee, I don't want to learn another package manager" -- but conda / anaconda.org is extremely worth it. It really is a major engineering step forward from the existing package deployment strategies in Python.

I even configure my travis.yml CI scripts to download Miniconda, create a conda environment from a requiremenets.txt, and then build and test my code via conda on the contiguous integration VM itself.

The only worry is how strongly tied conda and anaconda.org are to the future of Continuum. Given how much Continuum speaks of open-source work, one would hope that these projects essentially live independently (or that forks of them would) but you never know. I do admit that is a major downside.

Re:Python, Machine Learning, and Language Wars (2015) by stared on Tuesday March 22, 2016 @01:32PM (#11338254)
I like Anaconda (and I recommend it as the easiest installation for data sci), but on OS X it is easy to install Python and relevant numerical packages with Homebrew and pip.
Re:Python, Machine Learning, and Language Wars (2015) by drauh on Tuesday March 22, 2016 @01:37PM (#11338299)
Currently, pip install of numpy (and scipy) does not pick up MKL, which you would want if you do lots of linear algebra and FFTs.
Re:Python, Machine Learning, and Language Wars (2015) by a3n on Tuesday March 22, 2016 @07:56PM (#11340928)
I have anaconda installed at work/windows and home/linux. One minor annoyance is that it doesn't play well (at all?) with virtualenv. They worked so hard to get a huge suite of off the shelf third party libraries integrated, it's curious that they would come up with a replacement for virtualenv.

I'm halfway hoping that I don't know what I'm talking about.

Re:Python, Machine Learning, and Language Wars (2015) by spdustin on Tuesday March 22, 2016 @08:16PM (#11341044)
Re:Python, Machine Learning, and Language Wars (2015) by a3n on Tuesday March 22, 2016 @08:37PM (#11341163)
Right, I know about that. What I'm pointing out is that anaconda doesn't support http://docs.python-guide.org/en/latest/dev/virtualenvs/

It's nice that they have an env tool, but why didn't they just use the existing virtualenv?

Re:Python, Machine Learning, and Language Wars (2015) by spdustin on Tuesday March 22, 2016 @08:50PM (#11341246)
Because using theirs means not having to recompile numpy (for example) with each environment.

Why doesn't anaconda's env work for you?

Re:Python, Machine Learning, and Language Wars (2015) by a3n on Tuesday March 22, 2016 @09:42PM (#11341488)
Not saying it doesn't work, just wondering why the duplication of functionality. Which you answered, thanks.
Re:Python, Machine Learning, and Language Wars (2015) by spdustin on Tuesday March 22, 2016 @11:11PM (#11341854)
conda env also installs other dependencies for code. Really, the whole point of it all is that conda replaces pip and virtualenv.

Conda manages more than just Python packages, but other dependencies as well. For example, here's list of packages in a new environment I created with

  $ conda create -n hn python
  $ source activate hn
  $ conda list -c

  openssl-1.0.2g-0
  pip-8.1.1-py35_0
  python-3.5.1-0
  readline-6.2-2
  setuptools-20.3-py35_0
  sqlite-3.9.2-0
  tk-8.5.18-0
  wheel-0.29.0-py35_0
  xz-5.0.5-1
  zlib-1.2.8-0
setuptools and wheel are the only Python packages installed. The others are non-Python packages that conda is managing for my environment.

For all the talk of vendoring in package manager tools these days, I really, really like the way conda manages this stuff.