> The Python Foundation needs to adopt or create standard and supported ways of doing things

The Python community has adopted or created a standard way of doing things for the (checks calendar) fourth or fifth time now.

Right, but the community is different than the foundation. I would like an authoritative "this is how you do it" doc, from the foundation, about setup/packaging/dist, and I want it to work. And if said doc advises using some third party tool, then the foundation needs to fund or take over maintenance of that tool so that it continues to work and the "standard" way of doing it doesn't change every year.

It seems crazy to me that we don't have this yet.

There have been like three or four iterations of the "here's the official way to package and distribute python code" document. Here's the latest: https://packaging.python.org/

I've seen that before, but I dunno, it seems kind of anti-python[0] to have an "official doc" that basically says "here's a third-party tool we like, but also here's 5 other tools you could use instead." I want ONE tool, built and maintained by the same people who build and maintain python, so that it always works.

[0]: There should be one-- and preferably only one --obvious way to do it. (from the zen of python)

That’s never been true for packages. Just because someone else did it doesn’t mean you can’t do it better. The goal in the last few years has been to make it possible for anyone to write a build system or tooling, just like any other library in Python. Otherwise there is no innovation.

Innovation or invitation to shave a Yak?

My use case - distributing a Python package containing Python/C extensions - is described as "Incomplete / Last Reviewed 2013-12-08".

I maintain a library implemented mostly in python with some simple cpython extension modules for performance. Since you can't run unit tests against an extension module before you've built it, my Makefile invokes setup.py just to get things compiled:

  python setup.py build_ext --inplace

I don't see a valid replacement for this invocation, but fortunately, it's the only one.

When I include the library from a project that uses it, the same `pip install' command that brings the library in also performs all the compilation described by my setup.py.

All I'd need to move away from deprecated syntax entirely is a way to build the module when I'm testing it within it's own source directory. Until then I won't be changing anything.

It looks like setuptools can handle Cython - https://setuptools.pypa.io/en/latest/userguide/distribution.... .

It's less crazy than you think. The foundation historically never had developer employees (today has just 1 - a CPython core developer that started 3 months ago). The only way taking over a project and making it de facto standard would be to have (IMHO at least) 5 full time employees working on it. That's a big investment the foundation doesn't have and no corporation commited to support that (for at least 3-4 years). Also, there's the huge backlash the PSF would have to deal with from people who inevitable don't like the choosen standard.

I used to recommend articles with titles like "modern Python package structure and tooling" to graduates. I stopped because the tools (poetry, pipenv etc) were changing so much. I try to fit as much of our workflows into hand made templates and PyCharm now.

There was a post here a while ago about packaging or virtual environments. Within a few minutes, there were quite a few responses about better ways to do it.

Unfortunately, each response had a different recommendation. I wish I could find that thread again.

Maybe this one? https://news.ycombinator.com/item?id=26733423

We have been talking about how to support customers who want to use python. At least for me it isn't very appealing, seems like it could take a lot of attention to keep up with what's expected.

The difference between the community adopting a "standard", and something becoming the standard/being included into the langage by a direct decree is enormous. Especially with Python's philosophy.

Obligatory https://xkcd.com/927/

Also https://xkcd.com/1987/

^THIS^

Nowadays I use Poetry and PyInstaller, which make things easier.

For some of my projects, I produce a "compiled"/bundled executable that takes all the pain out of distribution. Here's my config, maybe it will be useful to you:

https://gitlab.com/stavros/harbormaster/-/blob/master/.gitla...

I’ve noticed Poetry being used more and more for projects I setup locally, and I must say, it does seem to be very, very good. Minimal overhead, easy to adopt, etc. I’ll need to setup a proper workflow with it for my current projects. Being able to abstract dev/prod environments in particular are really handy.

I'm sort of the Python hero at work, and, for me, the one supremely magical thing that Poetry brings to the table is that I am finally able to spec out a Python development workflow that I can easily document and explain to team members who don't already have a lot of detailed Python expertise.

Maybe the single most underappreciated thing about Poetry compared to other options is its tendency to fail safe.

Agreed, I really like it. It does everything and does it well, zero complaints so far.

I managed to find an annoying situation with poetry in which if you specify a range of versions for a dependency, and one of your other dependencies specifies a slightly different range of versions for that dependency, it will refuse to proceed. (I was expecting it to have chosen something in the intersection of the ranges, but it did not)

Hm, yeah, that definitely sounds like a resolver bug. Did you report it? I'm curious to see if it was fixed.

I suspect in GP's case there was something blocking the intersected range. That should work perfectly.

The poetry author doesn't want to support package version range overrides like yarn does [1]. His call I suppose, but I think it's the one major flaw in the otherwise giant step that poetry takes forward.

[1] https://github.com/python-poetry/poetry/issues/697

I did not report it because I found somebody else had already reported it and it was a WONTFIX (for reasons I don't immediately recall).

Poetry doesn't play nice with PyTorch and many other ML libraries.

We use it for both pytorch and tensorflow (sometimes in the same package) and it works fine.

How do you manage cuda versions of pytorch? At the moment I'm using poethepoet to install them separately but it feels very suboptimal.

So I personally didn't do the sysadminning, but it's a docker container in which CUDA has been loaded, and after that it 'just works'.

One of the nice things about Conda is that it will handle these kinds of dependencies for you within the environment it sets up.

While it doesn't really matter for web/scripting use cases, it's an absolute godsend for DS/ML workflows.

But the resolver, oh why does the resolver take so long?

we do conda -> mamba for that exact scenario :)

I've stopped recommending Python for learning programming, and instead have started recommending Go.

Python's environment difficulties make it exponentially harder for beginners to focus on actual programming concepts. Things like pyenv or virtualenvs or any of those things are blackboxes to beginners.

Go is simple, it's tightly coupled to version control so you can teach git at the same time in a meaningful way, and it's low level enough that CS concepts come up frequently, while being powerful enough that it will scale with a beginner's knowledge.

My wife started to learn to program - the setup overhead in python (going via mac -> git -> windows / linux) was crazy.

Can you not use your system's default python and open idle? Why involve git?

On Mac, at least as of Catalina, the system default Python is 2.something.

You can't just use brew?

Sure, but you specifically asked about the system's default Python.

Coming from Java, my reaction to pyenv and virtualenv was "why is this even needed?" Package management and SDKs are perennial fun times for all languages, but pyenv and virtualenv were just weird.

Unfortunately Go does not have an equivalent to Pandas as far as I know. The biggest attraction of Python is the ecosystem.

If you have a specific library or platform requirement then this advice is obviously not applicable, but for learning programming and software engineering principles, I honestly can't think of a better introductory language than Go.

I love Python, and the ecosystem is vast, you're right, but it has a serious issue with packaging, delivery, and dependency management. I have almost a decade of software engineering under my belt, and Python environment issues give me a hard time _regularly_. For beginners, it is a major demotivating factor that can be a death sentence for learning.

Interesting. Do you have a beginner’s course or tutorial that you would suggest? Something like the Rust book is wha came to mind

I would start with https://gobyexample.com/ and https://learnxinyminutes.com/docs/go/

Go By Example will take a beginner from Hello World to writing a server in a short time, and you can naturally splice in other skills.

> Python is such a joy to write and such a nightmare to set up, package and distribute.

Here's my simple-minded metric on when this problem can be called "solved":

Get a VPS from GoDaddy.

Deploy your Django app as easily as you can deploy a PHP app, say, Wordpress.

I have written about this before. I love Django. And yet I think that they have done a huge disservice with the development server and DB configuration out of the box. Even for a simple application, going from developing on your desktop to deploying the same application on, as an example, GoDaddy, is in a range between nightmare and impossible. I have personally given up multiple times and just said "Fuck it! Just use WP" even when I really, truly wanted to stay in the Python/Django ecosystem.

Try it. Develop a simple photo album application. Get a VPS from GoDaddy and deploy it.

No Heroku et. al., are not solutions. They are indicative of the problem.

I 100% agree. WordPress is the gold standard for ease of deployment. I specifically picked WordPress over other solutions because I knew the website will need maintenance without my help, and WP is simple enough for that.

if you can use docker, that solves pretty much all the python packaging mess and makes deploying your challenge of deploying random python application to a random VPS pretty simple.

i'm not saying that python packaging isn't a mess. but it can be largely mitigated.

As someone else said, Docker introduces yet another level of crazy to having to deploy a Python/Django app. I've looked into and used all of these technologies. We have developed internal "How To" documents on how to deploy different types of apps into a variety of environments. Reading through the Python/Django "How To" is a sobering experience.

This coming from someone who would truly like to use Python/Django exclusively. I hate, hate, hate telling people to just use WP or go a different route. I do not enjoy touching PHP. And yet...

Docker takes away some problems and creates other, smaller problems. It also results in more complexity. That complexity is a problem.

I can teach a slightly technical person to deploy WordPress on a shared host with a pre-installed shared LAMP stack. I cannot do this with Docker, not to to the extent it's sustainable without ongoing help from me.

Sorry, but GoDaddy is a disgrace, and your comment makes no sense at all, you are comparing WP with Django. And instead of recommending VPS as a solution, you recommend the worst party offering these services?

GoDaddy should burn in hell for their shady shit.

GoDaddy is a disgrace? It's shady shit?

C'mon, get some perspective.

I have been using GoDaddy for, well, I forget how long, maybe two decades. I have run websites for multiple companies from their servers. I can't remember a single issue. Not one. Even hosting and managing email. Frankly, I don't know what you are talking about.

I have also used Linode, AWS, Dreamhost, Rackspace and a bunch of others for different kinds of work. Not sure where the hatred for GoDaddy comes from.

I also know a bunch of people using GoDaddy servers. I think people repeat shit just because they think they sound "cool" and yet have no clue what they are talking about. Also, GoDaddy support, on the rare occasions they are needed, has always been top notch. So, yeah, no clue what you are talking about.

GoDaddy is shady and I would not recommend it ever, but they are a good representative of the target market - something people pick despite the shadiness and inflexibility, because they require zero technical competency to get something up and running.

I almost wonder if we need something like deno but for the python language--a reboot that uses the same language and syntax, but jettisons the cruft of packaging, etc. and starts over with a better developer experience. Perhaps just reference and import packages directly by URLs and forget all the pain and struggle of curating system-wide and virtual environments.

> Python is such a joy to write and such a nightmare to set up, package and distribute.

Agreed. Though I would not call it a "nightmare," the python ecosystem and runtime are cumbersome and annoying to work with when compared to some other languages that have put in a lot of thought into DX.

Still, it could be worse.

Once you hit the sweet spot of developing for cross-platform (even just Linux, MacOS, and Windows) and supporting normal average-people users and have (even optional!) C dependencies, Python's packaging situation quickly deteriorates into "nightmare" territory.

This is exactly my problem. I have to support Windows (a locked down corporate version) and Linux.

> Python is such a joy to write

Try out Ruby: so many Python syntax decisions make no sense after you see the Ruby way.

I began my professional career writing Rails apps (previous experience in software was cobbled together Python and .NET scripts), and 1-2 years later made the full transition to Django and Python, mostly because my mentors and friends at the time were writing projects in Python. Plus, the job market for fullstack Python devs was much more lucrative at the time.

I must say though, I do miss Ruby and the syntactic sugar, packaging/dependency system, etc. The transition itself was also very painless. I’m keen on checking out the current state of Ruby and/or Rails 4-5 years later, as I’m sure much has changed!

The two main frameworks I use(d) have relative equivalents in both languages which is super nice (Flask and Sinatra, Rails and Django). Right now I’m in the FastAPI craze, but perhaps I’ll opt for Rails for my next personal project.

Honestly, I don't think you'll find it feels like much has changed at all - modern Rails is still pretty consistent with Rails 4+, modern Ruby is still pretty consistent with Ruby 2+. Its one of my favorite things about the language and framework combo, it feels like once we got past Ruby 1.9.3 and Rails 3 things have been pretty well locked in.

There are new niceties and improvements, but bundler is still bundler, everything is still an object, etc.

I did, but so many Ruby syntax decisions made no sense to me.

Im convinced the only people that like ruby are those that want to delve into metaprogramming. It might be fun for the person that constructs it but it is a nightmare as an outsider looking into a project.

I know this is more rails but the fact that autoloading is even a thing speaks to the wild nature of ruby.

This might just be me not understanding Ruby well enough, but managing packages and dependecies feel even worse in Ruby.

Conda has become vastly better since I used it a few years ago. Worth a try again

That’s a pretty good idea, also a tool many Python developers already use, making the move between the two languages easier.

how? In ruby you are supposed to use rubygems and bundler for dependencies, and things generally work fine, and it's been like that for a decade or so.

There have been a few version management tools (rvm, rbenv), but those were all "fine" and still are, AFAICT, and you don't strictly need them.

Meanwhile for inexplicable reasons in pythonland we keep reinventing package/version/environment management.

I had a friend who managed to mess up their local env this morning to the point of screwing up their OS install, and it reminded me of the XKCD on python env[0].

It's at least 3 years old, and things have not improved.

[0]https://xkcd.com/1987/

I used to love Ruby. But it’s too exciting. Python is boring, in a good way. Like mowing the lawn. Or buying potatoes. I no longer want excitement in the process, only in the results.

> Python is such a joy to write and such a nightmare to set up, package and distribute.

Python is such a joy to write and such a nightmare to maintain, set up, package and distribute.

Fixed that for you.

Agreed. There has been so much churn and fragmentation with build tools over the last few years. So much complexity for what is not such a complicated problem to solve. Hopefully this is the "has to get worse before it can get better" part.

> So much complexity for what is not such a complicated problem to solve.

As someone who actually worked weeks on their spare time, I beg to differ. This is a very complicated problem to solve. And everyone has their own opinion on how things should work, which is goverened by their narrow use case. But you see a "standard" packaging tool needs to be the opposite of narrow use case. The main reason Anacond Inc exists is because they wanted to solve this for data science. Even with them being a relatively big corporation their "solution" is not loved universally, but works ok most of the time.

I disagree. I actually think it’s a complicated problem because there’s so many issues that have to be solved all at once.

What has been happening is that someone writes a build tool to solve a specific problem that they have but they neglect to solve all the other ones. Why would they? They’re not being funded.

That’s what caused (and still causes) a lot of the churn in JS too.

setting up is still unsolved, but:

- for librairy packaging and general dep maangement, poetry is the solution

- for packaging a web project and ship it on the server, shiv is the solution

- for distribution, half the solution is nuitka for compiling the program. Creating the installer is now the hard part

I haven't looked at Nuitka in a long time. Last time I did Windows support was not great and compilation took forever. I'll try it again.