News for hackers. Stuff that matters.
Re:How we rolled out one of the largest Python 3 migrations by carlmr on Wednesday September 26, 2018 @03:25AM (#18073529)
They're also fast and small, which while not always a big matter is very satisfying for me.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @08:44AM (#18075328)
I work in a Python shop. The Docker images we build are nearly 1 GB. I just built a Go service whose image is only 2.5MB. Admittedly it’s much simpler than the Python apps, but even a complex Go app would never reach the size of our Python app for a number of reasons:

1. Python apps require a distro base image while Go can run on scratch

2. Python images ship with the full standard library; not just the bits you import

3. In Python, if you add a dependency only to use 1 function or variable, you still end up with the whole dependency in your Docker image, while I’m pretty sure Go’s linker strips unused code.

Re:How we rolled out one of the largest Python 3 migrations by carlmr on Wednesday September 26, 2018 @10:21AM (#18076187)
I agree, I'm not a python fan. Python has a few good libraries I can't find in other languages, but it's slow, bad for multi core utilization, hard to distribute, is very wasteful with how many dependencies need to be included, has a terrible package manager and I prefer static strong typing.
Re:How we rolled out one of the largest Python 3 migrations by icebraining on Wednesday September 26, 2018 @03:57PM (#18079664)
Have you tried mypy? It's a pretty good static typing system for Python - arguably better than Go's.
Re:How we rolled out one of the largest Python 3 migrations by kartickv on Saturday September 29, 2018 @01:51AM (#18098907)
I'd like type annotations to be used to optimise performance. After all, if it has been statically verified that a particular variable is always an instance of class X, why not use that to optimise code?

This is an argument for type annotations to be integrated into every dynamically typed language, rather than tacked on via an external tool.

Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @05:30PM (#18080451)
TL;DR - I continue to root for Python's typing story, but it's just not there yet.

I have, and I wanted to like it. On its face it seems like it should be a lot better than Go's--after all, it supports generics and union types! But it falls over in trivial cases, like recursive types (i.e., there's no way to model tree structures such as JSON or linked lists). A few other hard/impossible/confusing things come to mind:

1. How do you declare a typevar for a certain scope. If I define a type parameter `T` for function `foo`, I only want `T` to be scoped to `foo`. I don't want the type checker getting confused with `T`s for other functions/classes/etc. 2. What is the signature for a function that takes args/kwargs? 3. It straight up doesn't work with popular libraries like SQLAlchemy (last I checked, these were simply not supported because the likes of SQLAlchemy are "too magical"--this is a fair take, but frustratingly limiting for users).

These are just a few because my memory is poor, but I run into these sorts of things by the dozens every time I try to use mypy. It's just not ready for prime time. Go's type system is limiting, but its limitations are much more predictable and even less limiting (it turns out recursive types and poor-man's union types are quite a bit better than first-class, non-recursive union types, for example).

Re:How we rolled out one of the largest Python 3 migrations by carlmr on Thursday September 27, 2018 @03:14AM (#18082582)
It's also tacked on. It's a bit too optional. If one team member doesn't care about typing he makes all his colleagues do the work to make his code work with mypy.
Re:How we rolled out one of the largest Python 3 migrations by icebraining on Wednesday September 26, 2018 @03:47PM (#18079587)
While it can't compete with 2.5MB, python:3.6-alpine (which includes points 1 and 2) weighs less than 100MB. You need a lot of Python code to get to 1GB.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Wednesday September 26, 2018 @06:26PM (#18080767)
Fair point. Our largest Python image has only 255 Mb of Python dependencies and ~50 Mb of source code. If we could use alpine (our compliance auditors strongly prefer centos base images), it would only be ~400 Mb. This is easily an order of magnitude bigger than an equivalent Go program, but still quite a lot better.
Re:How we rolled out one of the largest Python 3 migrations by icebraining on Wednesday September 26, 2018 @08:34PM (#18081376)
Is that really just Python code?? Must be over a million lines, no? I've worked with Odoo, which is a bit of a kitchen-sink (ERP, CRM, POS, sales, accounting, invoice, stock management, manufacturing control, website builder, marketing and a bunch more) and its Python code weighs just 15MB, the rest is JavaScript or data files.
Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Thursday September 27, 2018 @11:34AM (#18085465)
It's not just souce code--it's also docs and test code and other things that are tedious to omit given our current Docker image hierarchy and repository structure. Our docs are largely Sphinx docs in Python docstrings; a decent minifier could probably reduce this, but it's probably not worthwhile for a ~5% improvement on overall image size.
Re:How we rolled out one of the largest Python 3 migrations by carlmr on Thursday September 27, 2018 @03:13AM (#18082579)
My favorite Python library is pandas. If I only include this with pyinstaller I already get a >500MB executable.

IMHO unacceptably large, but that's because Python doesn't know which parts of pandas it might need to execute the program.

Re:How we rolled out one of the largest Python 3 migrations by icebraining on Thursday September 27, 2018 @05:31AM (#18083086)
Code pruning is a problem in Go too; as soon as your codebase or any of your dependencies use reflection, it gets disabled for the whole binary.

A simpler alternative would be for Numpy and Pandas to provide its features as subpackages, like Airflow does: https://airflow.readthedocs.io/en/latest/installation.html

Re:How we rolled out one of the largest Python 3 migrations by weberc2 on Thursday September 27, 2018 @11:17AM (#18085307)
I’m almost certain that only applies to individual compilation units and not the whole AST. In other words, if I use reflection in my main package, code pruning still works on dependencies, which is quite a lot better than the Python situation.