News for hackers. Stuff that matters.
Re:How we rolled out one of the largest Python 3 migrations by mr_toad on Tuesday September 25, 2018 @07:58PM (#18071564)
> However, rather than use the native toolchains directly, such as Xcode for macOS, we delegated the creation of platform-compliant binaries to py2exe for Windows, py2app for macOS, and bbfreeze for Linux.

I wish the authors of more Python tools would deploy standalone applications. I do not like having to maintain various sets of Python installers/package managers (because every Python tool seems to use a different installer). Especially on cloud servers that often lack a whole set of dependencies that Python developers just seem to take for granted.

I’m not a Python developer. I don’t have the time or the inclination to repackage various tools and untangle dependencies.

Given the choice between trying to figure out how to get multiple Python tools to behave together, or using another tool, I’ll almost always choose an alternative.

Re:How we rolled out one of the largest Python 3 migrations by cjhanks on Tuesday September 25, 2018 @08:56PM (#18071820)
This is possible and actually pretty trivial.

You install all 3rd party dependencies into some directory. The command line entry point is then a simple BASH script which sets the PYTHONPATH to the appropriate installation location and then does the appropriate exec call.

You then have a functionally portable python installation.

Re:How we rolled out one of the largest Python 3 migrations by lugg on Tuesday September 25, 2018 @10:13PM (#18072156)
Pretty trivial?

Really?

I think as soon as you start writing all caps trivial has gone right out the window.

Trivial would be application dependencies managed by the system package manager.

I'm at the point where I won't touch a python app or library that can't be installed via Pacman. It's just not worth my time.

And... python is still the only thing on arch that gives me the shits everytime it upgrades.

Re:How we rolled out one of the largest Python 3 migrations by throwawaymath on Wednesday September 26, 2018 @12:39AM (#18072833)
This exchange is a good demonstration that the word "trivial" has lost its meaning in the same way "literal" has. Much like I usually hear someone use term "literally" for figurative emphasis, these days I mostly hear the word "trivial" used to describe something which is clearly nontrivial.

Math textbooks have been doing this for decades, but it's leaked into common parlance with online discussion.

Re:How we rolled out one of the largest Python 3 migrations by ezequiel-garzon on Wednesday September 26, 2018 @02:15AM (#18073187)
I believe in the mathematical community the potentially offensive term for outsiders is non-trivial, used in a technically correct manner, but oftentimes applied as synonymous of epically hard, which unexpectedly throws people off, especially the author of that non-trivial work!
Re:How we rolled out one of the largest Python 3 migrations by CogitoCogito on Wednesday September 26, 2018 @03:44AM (#18073657)
I think the best translation of "trivial" when used amongst mathematicians is that it is something you should be able to figure out with your current knowledge without too much difficulty (though it might require an hour of thought). Said in another way, you don't need to learn/develop new tools or techniques for something that is trivial.

Of course this is not when most people think when they hear the word so really it's a term of art that should probably be avoided when talking to non-mathematicians.

Re:How we rolled out one of the largest Python 3 migrations by Ntrails on Wednesday September 26, 2018 @04:03AM (#18073766)
> I think the best translation of "trivial" when used amongst mathematicians is that it is something you should be able to figure out with your current knowledge without too much difficulty

As i read through my old uni maths notes there are often wild leaps from a to e along with a little scrawl saying "trivially" or "obviously". They may have been true once, but god dammit 21 year old me was a knobber

Re:How we rolled out one of the largest Python 3 migrations by cjhanks on Wednesday September 26, 2018 @11:41AM (#18077069)
It takes approximately a year to learn to walk after you are born. But almost everybody figures it out. And once they know, they practically never forget. So I don't know, is it not trivial?

I have never had to support all of the mobile environments of DropBox nor the scale, so I cannot claim that my 99% solution would ever meet their 99.999% requirements. But I have been able to package Python apps for Mac OSX, WinVista, Win7, Ubuntu, and CentOS at the same time using that strategy.

Re:How we rolled out one of the largest Python 3 migrations by NLips on Wednesday September 26, 2018 @04:33AM (#18073940)
Being charitable, I think the parent means that the python developer sticks all the dependencies in a directory and creates a bash script to set PYTHONPATH and launch. The user receives a directory rather than an executable, but only has to use the bash script, rather than worry about any of the Python in the directory.
Re:How we rolled out one of the largest Python 3 migrations by Already__Taken on Wednesday September 26, 2018 @06:20AM (#18074479)
Now your lovingly created cross-platform app is back to linux only because you used bash to invoke it.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @08:33AM (#18075240)
Use a Python script instead.
Re:How we rolled out one of the largest Python 3 migrations by oblio on Wednesday September 26, 2018 @10:02AM (#18075995)
Nah, he should use some turtles instead.
Re:How we rolled out one of the largest Python 3 migrations by orf on Tuesday September 25, 2018 @09:03PM (#18071844)
Isn't this just a hacky virtualenv? 'python3 -mvenv dir && ./dir/bin/pip install requirements'?
Re:How we rolled out one of the largest Python 3 migrations by jofer on Tuesday September 25, 2018 @09:43PM (#18072030)
There are a lot of minor differences, but the biggest difference is that you're able to be completely independent of the system python install. You bundle a complete python interpreter, all libraries needed, etc.

The user doesn't need to have python installed at all, and if they have 2.x instead of 3.x or 3.3 when you're expecting features that are only present in >= 3.5, it's no issue.

This may sound trivial, but it's a _huge_ deal, particularly when you need to deploy something that runs on multiple different OSes and versions of OSes.

Other than that, the "directory full of libs, binaries, and code" approach is a lot easier to package into something that will work well with the native package manager (e.g. an .msi for windows, etc).

Re:How we rolled out one of the largest Python 3 migrations by rtpg on Tuesday September 25, 2018 @11:36PM (#18072556)
in the ideal, those making the tools would provide this packaging, instead of asking all users to have `pip` properly set up. Similar issues exist w/r/t npm in my opinion.
Re:How we rolled out one of the largest Python 3 migrations by oweiler on Wednesday September 26, 2018 @03:00AM (#18073407)
That's why most of the tools I use in my day to day work are Go or Rust binaries.
Re:How we rolled out one of the largest Python 3 migrations by carlmr on Wednesday September 26, 2018 @03:25AM (#18073529)
They're also fast and small, which while not always a big matter is very satisfying for me.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @08:44AM (#18075328)
I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons:

1. Python apps require a distro base image while Go can run on scratch

2. Python images ship with the full standard library; not just the bits you import

3. In Python, if you add a dependency only to use 1 function or variable, you still end up with the whole dependency in your Docker image, while I’m pretty sure Go’s linker strips unused code.

Re:How we rolled out one of the largest Python 3 migrations by carlmr on Wednesday September 26, 2018 @10:21AM (#18076187)
I agree, I'm not a python fan. Python has a few good libraries I can't find in other languages, but it's slow, bad for multi core utilization, hard to distribute, is very wasteful with how many dependencies need to be included, has a terrible package manager and I prefer static strong typing.
Re:How we rolled out one of the largest Python 3 migrations by icebraining on Wednesday September 26, 2018 @03:57PM (#18079664)
Have you tried mypy? It's a pretty good static typing system for Python - arguably better than Go's.
Re:How we rolled out one of the largest Python 3 migrations by kartickv on Saturday September 29, 2018 @01:51AM (#18098907)
I'd like type annotations to be used to optimise performance. After all, if it has been statically verified that a particular variable is always an instance of class X, why not use that to optimise code?

This is an argument for type annotations to be integrated into every dynamically typed language, rather than tacked on via an external tool.

Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @05:30PM (#18080451)
TL;DR - I continue to root for Python's typing story, but it's just not there yet.

I have, and I wanted to like it. On its face it seems like it should be a lot better than Go's--after all, it supports generics and union types! But it falls over in trivial cases, like recursive types (i.e., there's no way to model tree structures such as JSON or linked lists). A few other hard/impossible/confusing things come to mind:

1. How do you declare a typevar for a certain scope. If I define a type parameter `T` for function `foo`, I only want `T` to be scoped to `foo`. I don't want the type checker getting confused with `T`s for other functions/classes/etc. 2. What is the signature for a function that takes args/kwargs? 3. It straight up doesn't work with popular libraries like SQLAlchemy (last I checked, these were simply not supported because the likes of SQLAlchemy are "too magical"--this is a fair take, but frustratingly limiting for users).

These are just a few because my memory is poor, but I run into these sorts of things by the dozens every time I try to use mypy. It's just not ready for prime time. Go's type system is limiting, but its limitations are much more predictable and even less limiting (it turns out recursive types and poor-man's union types are quite a bit better than first-class, non-recursive union types, for example).

Re:How we rolled out one of the largest Python 3 migrations by carlmr on Thursday September 27, 2018 @03:14AM (#18082582)
It's also tacked on. It's a bit too optional. If one team member doesn't care about typing he makes all his colleagues do the work to make his code work with mypy.
Re:How we rolled out one of the largest Python 3 migrations by icebraining on Wednesday September 26, 2018 @03:47PM (#18079587)
While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @06:26PM (#18080767)
Fair point. Our largest Python image has only 255 Mb of Python dependencies and ~50 Mb of source code. If we could use alpine (our compliance auditors strongly prefer centos base images), it would only be ~400 Mb. This is easily an order of magnitude bigger than an equivalent Go program, but still quite a lot better.
Re:How we rolled out one of the largest Python 3 migrations by icebraining on Wednesday September 26, 2018 @08:34PM (#18081376)
Is that really just Python code?? Must be over a million lines, no? I've worked with Odoo, which is a bit of a kitchen-sink (ERP, CRM, POS, sales, accounting, invoice, stock management, manufacturing control, website builder, marketing and a bunch more) and its Python code weighs just 15MB, the rest is JavaScript or data files.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Thursday September 27, 2018 @11:34AM (#18085465)
It's not just souce code--it's also docs and test code and other things that are tedious to omit given our current Docker image hierarchy and repository structure. Our docs are largely Sphinx docs in Python docstrings; a decent minifier could probably reduce this, but it's probably not worthwhile for a ~5% improvement on overall image size.
Re:How we rolled out one of the largest Python 3 migrations by carlmr on Thursday September 27, 2018 @03:13AM (#18082579)
My favorite Python library is pandas. If I only include this with pyinstaller I already get a >500MB executable.

IMHO unacceptably large, but that's because Python doesn't know which parts of pandas it might need to execute the program.

Re:How we rolled out one of the largest Python 3 migrations by icebraining on Thursday September 27, 2018 @05:31AM (#18083086)
Code pruning is a problem in Go too; as soon as your codebase or any of your dependencies use reflection, it gets disabled for the whole binary.

A simpler alternative would be for Numpy and Pandas to provide its features as subpackages, like Airflow does: https://airflow.readthedocs.io/en/latest/installation.html

Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Thursday September 27, 2018 @11:17AM (#18085307)
I’m almost certain that only applies to individual compilation units and not the whole AST. In other words, if I use reflection in my main package, code pruning still works on dependencies, which is quite a lot better than the Python situation.
Re:How we rolled out one of the largest Python 3 migrations by Rotareti on Wednesday September 26, 2018 @04:18AM (#18073862)
I hope some day some big corp decides to write a decent (open source) Python to C compiler, which makes optional optimizations based on type annotations.

I find this project: https://github.com/Nuitka/Nuitka very interesting, but its written and maintained only by a single person and I never got it to work with any of my apps.

Re:How we rolled out one of the largest Python 3 migrations by fabioz on Wednesday September 26, 2018 @04:41AM (#18073983)
Cython is a pretty good Python to C compiler and in its latest release is using type annotations... the thing is, you shouldn't need to compile your whole program (compiling is really slow and interpreted Python is fast enough for most of the code).
Re:How we rolled out one of the largest Python 3 migrations by goerz on Wednesday September 26, 2018 @08:30AM (#18075205)
I was under the impression that Cython is a different language (a subset / dialect of Python). So, not a Python to C compiler.
Re:How we rolled out one of the largest Python 3 migrations by kartickv on Saturday September 29, 2018 @01:54AM (#18098920)
Good point. I never understood why the developers of every language don't make it trivial to build a .exe or .app file that users can double-click to run. Seems like the Python team is penalising developers for using Python :)

It can internally use an interpreter, JIT, bytecode or full native code like C. That's a different discussion. Just don't make it a pain to distribute.

Re:How we rolled out one of the largest Python 3 migrations by kccqzy on Wednesday September 26, 2018 @12:41AM (#18072843)
Even when I am a python developer, there is still a difference between (a) this dependency that I explicitly rely on and that I need to sort out packaging issues for, and (b) code I treat as a black box and simply expect to work.

More often than not, a typical developer's python environment tends towards https://xkcd.com/1987/

Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @08:46AM (#18075347)
That xkcd is missing virtualenv, pyenv, pipenv, venv, and poetry.