News for hackers. Stuff that matters.
Re:Why we switched from Python to Go by disconnected on Tuesday October 17, 2017 @08:58PM (#15496177)
> conflicting system level packages

In Python, if you have control over the target system, this is solved with virtualenv. You can install whatever versions of libraries you want in there with pip without causing conflicts with system level packages.

If you intend to ship to end-users, yeah, you are kinda screwed. You are stuck using one of the various "freeze" methods, which, in my experience, kinda blow.

Re:Why we switched from Python to Go by weberc2 on Tuesday October 17, 2017 @11:37PM (#15496717)
We use Nix which is much nicer than pyenv/virtual environment, but it's still a pain compared to Go. This is mostly due to Python's runtime model, and maybe also the tendency for Python applications to have very tall, broad dependency graphs.

Working with nix is also not very easy; docker might fare better here, but probably just a lateral move. In any case, Go outclasses Python on deployments. Also, my company is looking into compiling Python as a means of code obfuscation; Go compiles by default, which (modulo stripping) is probably enough obfuscation for our purposes.

Re:Why we switched from Python to Go by protomok on Wednesday October 18, 2017 @12:30AM (#15496894)
I've been evaluating Cython for both code obfuscation and for using C based extensions to improve performance. Cython is pretty nice, but shipping binary only Python extensions can be such a pain, especially dealing with 2-byte vs 4-byte unicode representation issues.

I need to try Go one of these days !

Re:Why we switched from Python to Go by ptx on Wednesday October 18, 2017 @03:55AM (#15497578)
> shipping binary only Python extensions can be such a pain, especially dealing with 2-byte vs 4-byte unicode representation issues

The issue with different Unicode builds was fixed 5 years ago, in Python 3.3:

https://docs.python.org/3/whatsnew/3.3.html#pep-393-flexible...

Re:Why we switched from Python to Go by protomok on Wednesday October 18, 2017 @07:46AM (#15498288)
Yes it's nice to see this is now fixed. In my case I need to support Python 2.7 though, and for Python 2.7 the only workaround I know of is shipping two binaries (2 byte and 4 byte encoding) then loading the correct module at runtime.
Re:Why we switched from Python to Go by mixmastamyk on Wednesday October 18, 2017 @03:56AM (#15497583)
> especially dealing with 2-byte vs 4-byte unicode

Wasn’t that solved years ago?

Re:Why we switched from Python to Go by djsumdog on Wednesday October 18, 2017 @03:12AM (#15497447)
Virtualenv is nice, but lately I've grown to use docker containers instead. You can use the official containers for go, python or ruby, feed it the gemfile or requirements.txt or whether, tag it, push it and you have a permanent snapshoted image.
Re:Why we switched from Python to Go by nilved on Wednesday October 18, 2017 @03:57AM (#15497587)
Virtualenv is not nice; it's a pragmatic yet laughable hack.
Re:Why we switched from Python to Go by JelteF on Wednesday October 18, 2017 @06:20AM (#15498002)
It's definitely a hack, but for development environments it's quite an effective one. Which is why we actually created a (more user friendly) clone for Go development. It's called Virtualgo, and it's mentioned near the end of the article as well. https://github.com/GetStream/vg
Re:Why we switched from Python to Go by hasenj on Tuesday October 17, 2017 @09:58PM (#15496353)
> this is solved with virtualenv

Except when you run into edge cases, which happens all the time.

Re:Why we switched from Python to Go by Rotareti on Wednesday October 18, 2017 @06:56AM (#15498115)
Are you using the newer venv package from the stdlib or the old virtualenv package from pypi? I have never encountered a single issue with the former.
Re:Why we switched from Python to Go by gruez on Tuesday October 17, 2017 @10:33PM (#15496464)
elaborate?
Re:Why we switched from Python to Go by jeffshek on Wednesday October 18, 2017 @12:06AM (#15496817)
Most of the time pip installs work fine, but once a while after some type of OS upgrade (namely, my experience has been MacOS), you run into odd compiler types of issues.

So it's why a lot of teams will use either Vagrant / Docker to setup local developer environments.

Re:Why we switched from Python to Go by 0xCMP on Wednesday October 18, 2017 @12:57AM (#15496980)
Can confirm that once upgrading macOS required using Docker to be productive, because some C-dep we had stopped working. We have developers using macOS 10.10 to avoid any issues (although they're likely gone by now, it's not worth the effort to figure it out anymore).
Re:Why we switched from Python to Go by jon_richards on Wednesday October 18, 2017 @02:41AM (#15497344)
I've occasionally had things spontaneously break. Most recently the cryptography package just stopped installing on deployment. Had to add a pip upgrade on deploy, which somehow prevented the AMI I was using from installing some of its requirements. Had to add those packages to my project requirements.

Also some of the data analysis packages don't work with virtualenv.

Re:Why we switched from Python to Go by dozzie on Wednesday October 18, 2017 @08:13AM (#15498421)
> Most recently the cryptography package just stopped installing on deployment. Had to add a pip upgrade on deploy, [...]

Erm... Why the heck are you running pip when deploying software? O_o It should be a build step, not a deployment step.

Re:Why we switched from Python to Go by hasenj on Wednesday October 18, 2017 @09:35AM (#15498977)
Most deploy scripts I've seen tend to do this. Which is insane in my opinion, but the python toolset encourage this kind of approach by making it the default easy thing to do.

That's one of the things I prefer about Go: the compiler can cross-compile, so I can compile the entire codebase on my osx machine and produce a single statically linked linux binary that I can just copy over scp to the server. Deployment becomes infinitely simpler.

Re:Why we switched from Python to Go by dozzie on Wednesday October 18, 2017 @10:09AM (#15499255)
> Most deploy scripts I've seen tend to do this. Which is insane in my opinion [...]

Also, most programmers don't know a thing about administering servers. Ubiquity doesn't mean that it's a proper way, as you clearly see yourself.

> [...] the python toolset encourage this kind of approach by making it the default easy thing to do.

It's not quite the fault of Python tooling in particular. It's really the easiest way in general. Similarly, the easiest way to deploy a unix daemon is to run it as root user, because the daemon will have automatically access to all files and directories it needs (data storage, temporary files, sockets, pidfiles, log files). Anything else that is easier to manage requires to put a non-zero effort.

> That's one of the things I prefer about Go: [static linking]

Static linking has its own share of administrative problems. It's not all nice and dandy.

Re:Why we switched from Python to Go by jon_richards on Wednesday October 18, 2017 @08:28AM (#15498540)
For a number of reasons, I haven't bothered to do a complex build/deploy process. I write in python, I freeze the requirements into requirements.txt, and I type "eb deploy". Anything else is overkill for me.
Re:Why we switched from Python to Go by dozzie on Wednesday October 18, 2017 @08:52AM (#15498697)
And then you have different lists of installed packages in your development environment and in the production and even between two different production servers (which can silently and/or subtly break your software), and you need to manually install all the C/C++ libraries and compilers and non-Python tools and whatnot, and the upgrade process takes a lot of time and can break on problems with PyPI and/or packages that went missing, and you can't even sensibly downgrade your software to an earlier version.

Yes, avoiding learning how to properly package and deploy your software is totally worth this cost.

Re:Why we switched from Python to Go by hasenj on Wednesday October 18, 2017 @11:19PM (#15505242)
Python does not really have a "build" step. _That_ is the problem.

I think Heroku also played a role in prolifirating the idea that you can just push your code (from a git repo) to a server, and the server will take care of deploying it. Which usually means the server will install all the dependencies from pip during the process.

Re:Why we switched from Python to Go by hasenj on Wednesday October 18, 2017 @04:04AM (#15497604)
When you build a piece of software on many layers of abstractions, something will eventually break and leak all the way up in a nasty way that is difficult to debug or fix.

I don't have specific examples in mind but I've had a lot of frustrations with virtual-env.

Re:Why we switched from Python to Go by foota on Tuesday October 17, 2017 @09:02PM (#15496191)
Solved as long as there aren't native dependencies not managed by pip, which there often are.
Re:Why we switched from Python to Go by joshuamorton on Tuesday October 17, 2017 @11:04PM (#15496589)
Wheels solved this problem in 2013.

For context, you can install opencv, tensorflow, ROS, matplotlib, and the entire scipy stack in a virtualenv, with no external dependencies, using wheels.

This means that you can generate images, train a machine learning algorithm on them, compare the results to conventional CV algorithms, and display them in an ipython notebook all from a venv. There's a huge amount of C++ and even qt integration in that pipeline, all isolated.

It won't be maximally performant (ie for massive training), but for that you'd want distributed docker deploys or similar anyway.

Re:Why we switched from Python to Go by deathanatos on Wednesday October 18, 2017 @12:55AM (#15496976)
Wheels doesn't completely solve this problem. For example, your example of matplotlib, IIRC, has a dependency on libblas: a C library. This dependency isn't captured in the wheel metadata, and it's up to you to install the right libblas on the host, or somewhere where it will get loaded. (Though honestly, this isn't usually an unmanageable level of complexity. But virtualenvs are not free from external dependencies always.)

IIRC, we also had a problem where a wheel failed to find symbols in a SO it was linked against. It turns out that the wheel worked fine in precise, but failed in trusty, and we ended up having to split our wheel repository because the wheel was specific to the Ubuntu release. (It seemed, at the time, that whatever SO it was linked against appears to have made backwards incompatible changes without changing the major.) There are rare cases like this where the wheel is unique to attributes of the host that the wheel metadata can't capture.

Re:Why we switched from Python to Go by viraptor on Wednesday October 18, 2017 @06:57AM (#15498118)
> For example, your example of matplotlib, IIRC, has a dependency on libblas: a C library

And with the wheel packaging, you're free to embed that library in the wheel that depends on it. You can also not do that and rely on the system libraries. The wheel provides you a way to do what you want, but doesn't force you to do it.

The are good reasons for either of those approaches, so I'd say wheel does solve the issue.

Re:Why we switched from Python to Go by joshuamorton on Wednesday October 18, 2017 @11:29AM (#15499937)
As another user mentioned, they do, if the library maintainers take the time to do things correctly. The package I'm describing does not link outside of the virtualenv, the matlplotlib, opencv-python, and tensorflow libraries include all necessary dependencies (although you have to use a non-default renderer in matplotlib, because it doesn't bundle all of them).

What you say is correct, virtualenvs are not free from external dependencies always, but correctly build wheels are. Wheels and virtualenvs aren't the same thing.

Re:Why we switched from Python to Go by sethammons on Wednesday October 18, 2017 @12:14AM (#15496849)
Wheels just broke something in our build pipeline. They removed support for Python 2.6 and started tossing errors. I was able to fix it by pinning Wheel which probably should have been done originally by who ever made the build utility, but it would have been a non issue with Go and a binary.
Re:Why we switched from Python to Go by joshuamorton on Wednesday October 18, 2017 @12:29AM (#15496890)
Well, python 2.6 was also EoL'd 4 years ago, so yes, if you're using a no longer supported piece of software and not pinning your versions, I'd argue that you're inviting issues.
Re:Why we switched from Python to Go by sethammons on Wednesday October 18, 2017 @01:15AM (#15497042)
Sure. But if the build utility was just some binary, then it wouldn't matter. If Go was abandoned by all maintainers tomorrow or they broke all the packages, the already built binary will still work.

Should someone have changed the Python build tool to be 2.7 or 3? Maybe if they were bored and new it was something that needed work or wanted to be good tech citizen. However, what really happened is that no one even knew what the tool was really doing, just that it was part of a suite of tools in a build process, and no one would have looked twice at it ever again had wheel not removed support for 2.6. /me shrugs.

Re:Why we switched from Python to Go by joshuamorton on Wednesday October 18, 2017 @11:26AM (#15499899)
>But if the build utility was just some binary

I mean it totally would if the binary dynamically linked against a file you didn't have on your system, which is exactly what happened with `wheel` (python 2.6 doesn't have OrderedDict, which wheel now uses).

Re:Why we switched from Python to Go by pwang on Wednesday October 18, 2017 @12:55AM (#15496977)
Have you actually done this? If so, on what platform(s)?
Re:Why we switched from Python to Go by joshuamorton on Wednesday October 18, 2017 @11:10AM (#15499737)
Linux, If you'd like, I'll post the pipfile for the repository.
Re:Why we switched from Python to Go by alfla on Wednesday October 18, 2017 @02:30AM (#15497302)
Could you elaborate on how ROS fits into this framework?
Re:Why we switched from Python to Go by joshuamorton on Wednesday October 18, 2017 @11:31AM (#15499967)
That's a different project, but I also use ros in a virtualenv. Its a bit weird because you end up installing ros both inside and outside of the venv (various ros command line tools only use /opt/ros/... python deps), but your actual nodes run in your virtualenv.

And ros doesn't even need to be a wheel fwiw, its pure python, but its also just a painful thing to deal with for many, so it was worth mentioning.

Re:Why we switched from Python to Go by SmirkingRevenge on Wednesday October 18, 2017 @12:41PM (#15500608)
You can pip install pyspark now too!