News for hackers. Stuff that matters.
Re:Pipenv: promises a lot, delivers very little by sametmax on Wednesday December 05, 2018 @06:25PM (#18613806)
I'm very glad we have the wheel and ensurepip now.

Yet, I think PyPA has not been taking the best decisions regarding Python packaging.

Your Kenneth story is not the only "weird event" in their history.

E.G:

Did you know that we don't need "pyproject.toml" at all ? That there is already a production ready plain text standard to replace setup.py ?

Did you know that this standard has been perfectly working for TWO YEARS with regular setuptools, is incredibly simple to use and completly compatible with the standard "setup.py stuff" workflow (and hence the whole legacy tool stack) ?

Yep. And nobody talks about it.

Let me (re)introduce...

Setup.cfg !

Oh, I know... Most people believes it's a useless file.

After all, the Python documentation seldom states to use it and for only one tiny option:

https://docs.python.org/3.7/distutils/configfile.html

But no. Setup.cfg is awesome !!

Put one line in setup.py:

    import setuptools; setuptools.setup()
And you can now put all the rest - yes, everything - in setup.cfg. It's perfectly documented:

https://setuptools.readthedocs.io/en/latest/setuptools.html#...

Not only it has been working since 2016, but it has fantastic goodies:

    version = attr: src.__version__ 
will include the version from __init__.py

     "options.data_files" 
replaces the MANIFEST

     "license =  file: LICENCE.txt" 
loads the licence from any text file

Try it, it just works. And you can "python setup.py sdist upload" as usual with it. You don't need any new tool.

Now why did the PyPA decide to forget about this and create a new format ? The short explanation in PEP 518 is a bad joke. And why does nobody talks about it ?

When I asked the PyPA, they told me they were too invested in the new project to stop now. I don't like this answer at all: we suffered enough with python packaging during the "distutils, eggs, etc" fiasco.

setup.cfg works. It works now. It's nice. It's compatible. It does what we need.

Use it. Talk about it. Write about it.

Make sure a lot of people knows so that tool makers and PyPA finally acknowledge that there is not need for the XKCD comics about standard to be true again.

Re:Pipenv: promises a lot, delivers very little by scrollaway on Wednesday December 05, 2018 @06:52PM (#18613991)
I love setup.cfg, I've used it for years and indeed pyproject.toml is useless given that setup.cfg has existed for frickin ever at this point (and is supported by a multitude of tools). TOML is nice but its support isn't even in the stdlib which makes it very awkward to use for a core file like that.

Here are some examples of my libs/apps using it in the real world, if someone needs references for how to use setup.cfg with an empty or near-empty setup.py:

https://github.com/jleclanche/python-bna/blob/master/setup.c...

https://github.com/dj-stripe/dj-stripe/blob/master/setup.cfg

https://github.com/jazzband/django-oauth-toolkit/blob/master...

https://github.com/jazzband/django-push-notifications/blob/m...

Edit: I see you mention attr: src.__version__. I personally prefer doing it the other way around, with the version defined in setup.cfg and a pkg_resources snippet in __init__.py (https://github.com/HearthSim/python-hearthstone/blob/master/...).

To be honest I wish __version__ were automatically defined like that (but more reliably). Do you know if this was discussed in a PEP?

Re:Pipenv: promises a lot, delivers very little by ATsch on Wednesday December 05, 2018 @08:10PM (#18614486)
I don't understand the pyproject.toml hate. pyproject.toml exists specifically so you can specify your build system. Without it, you are basically forced to use setuptools/distutils as is currently the case. Hence, pyproject.toml and setup.cfg aren't at all in conflict.
Re:Pipenv: promises a lot, delivers very little by sametmax on Wednesday December 05, 2018 @08:39PM (#18614654)
- there is nothing you can do with pyproject.toml that you can't with setup.cfg. E.G: you are not forced to use setuptools to use setup.cfg. Any tool supporting pyproject.toml could support setup.cfg as easily, since it's a documented plain text format. It's a political decision.

- there are things you can't do with pyproject.toml you can with setup.cfg. E.G: you can't use pyproject.toml with legacy tools, or with just a fresh python setup. This may change in the future, but would requires a lot of effort because changing setuptools is a very tedious process.

- resources (time, people, money, documentation, public attention, communication, etc) invested in creating and supporting pyproject.toml could be invested in improving setup.cfg and its ecosystem support. E.G: Why does poetry support pyproject.toml and not setup.cfg ? No technical reason. Why does nobody knows this easy way to package python libs ? No technical reason reason either.

So not only the new format brings nothing on the table, but it is also a setback, AND add clutter to a situation that was just begining to be solved. It's not just poor engineering, it's poor manners really.

I've been coding in Python for 15 years. I've lived this: https://stackoverflow.com/a/14753678

Stop the pain.

Re:Pipenv: promises a lot, delivers very little by pseudalopex on Wednesday December 05, 2018 @11:36PM (#18615453)
The format of setup.cfg is whatever configparser accepts, which is different in different versions of Python.
Re:Pipenv: promises a lot, delivers very little by sametmax on Thursday December 06, 2018 @03:04AM (#18616236)
First, it managed to work fine for setuptools for 2 years accross all recent python versions. That's because the differences are minor and edge cases. Try to use pyproject.toml in most CI toolchain just for fun...

Second, the format of setup.cfg is defined in a documentation already, so there is a reference outside of configparser. Yes, the low level format is not explicitly defined (although it is implicitly): so let's do define it instead of creating a new one.

Third, it's still a much easier and saner task to rafine the definition of setup.cfg than to create a new standard. I don't even understand how this is controversial, espacially among people in the computing world, where we had those kind of problems for decades and we know the pros and cons, and consequence of this.

The "I add my little format because it's pure and better and current things suck" falacy is such a stereotype we should all be able to recognize it from miles away from nowaday.

Re:Pipenv: promises a lot, delivers very little by pseudalopex on Thursday December 06, 2018 @05:10PM (#18622841)
Using pyproject.toml just requires a recent version of pip. Which CI tools can't handle that?

I wouldn't call comments an edge case. The distutils documentation has a definition for comments, but I think it actually just uses configparser. setuptools just uses configparser. The pbr documentation has a slightly different definition, but I wouldn't be surprised if it just uses configparser too.

They also have different definitions of non-string types.

Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?

Re:Pipenv: promises a lot, delivers very little by sametmax on Friday December 07, 2018 @03:43AM (#18626040)
> Using pyproject.toml just requires a recent version of pip. Which CI tools can't handle that?

The most used one in the world: https://github.com/travis-ci/dpl/issues/822

Also the last time I used tox, anything complex didn't work either.

> Even if you call those edge cases, do you think a PEP that turned edge cases into silent errors would be approved?

Well the current PEP decided to turn a packaging situation that was stable into one that was not, again, after 15 years of mess with many versions of things. So you tell me.

Check the usage stats I posted in an other comment to see the problem.

Besides, yes, we do make compromise on best practices to allow peaceful transition all the time in Python. `async/await` allowing to be a variable silently. Non utf8 defaut encoding argument in open() in windows. Then... we fix it later.

Because I think you conveniently skip a lot of things I wrote in my comments. I clearly state that we would and should consider setup.cfg as a version 1 of the format. Then we would increment on that. I gave a detailed procedure on one way to do that, and there are others.

The point is, all your concerned can be addressed with a progressive transition, starting from setup.cfg. Actually we could even end up with a toml format in setup.cfg, __on the long run__, that matches exactly the current one.

While you addressed non of ours concerns. Just reject them. No will to even recognize there is a problem. It's insulting, really.

We did that during the 2/3 transition. Didn't work so well, did it ?

Re:Pipenv: promises a lot, delivers very little by toyg on Thursday December 06, 2018 @11:13AM (#18619249)
> we should all be able to recognize it from miles away from nowaday.

Oh but we do. Then we rationalize it away, because "this time...". Like we do for Big Rewrites.

It might have something to do with the fact that programming is mostly a craft you learn by doing it, so we overvalue "doing it again" because that's how we usually get better.

Re:Pipenv: promises a lot, delivers very little by scrollaway on Wednesday December 05, 2018 @08:26PM (#18614593)
They're not in conflict. The problem is pyproject.toml is superfluous. See my comment here: https://news.ycombinator.com/item?id=18614058
Re:Pipenv: promises a lot, delivers very little by sametmax on Wednesday December 05, 2018 @07:08PM (#18614094)
I like the way you do it too, and I don't think it needs any improvements besides being shared more.
Re:Pipenv: promises a lot, delivers very little by gvalkov on Thursday December 06, 2018 @04:52AM (#18616658)

  > Use it. Talk about it. Write about it.
I have a project that converts basic setup.py files to setup.cfg files [1].

Still happily using plain setuptools for library development and pip-tools for application development.

[1]: https://github.com/gvalkov/setuptools-py2cfg

Re:Pipenv: promises a lot, delivers very little by sametmax on Friday December 07, 2018 @03:45AM (#18626047)
Very cool. I'll share that.
Re:Pipenv: promises a lot, delivers very little by pseudalopex on Wednesday December 05, 2018 @06:53PM (#18613995)
setup.cfg lets you configure setuptools with declarative syntax. pyproject.toml lets you replace setuptools with something else. The PEP explains why they didn't just reuse setup.cfg.[1]

[1] https://www.python.org/dev/peps/pep-0518/#sticking-with-setu...

Re:Pipenv: promises a lot, delivers very little by scrollaway on Wednesday December 05, 2018 @07:02PM (#18614058)
The problem is that this is yet another boilerplate file.

The reasoning is good, but we were just arriving to the point that every Python tool out there is either compatible with tox.ini, setup.cfg, or both (much like the JS ecosystem has tools reading from package.json).

Now we have both Pipfile and pyproject.toml on top of it!

For a language that prides itself on its stability and backwards compatibility (especially when compared to the JS ecosystem), we churn through boilerplate files harder than Google churns through instant messaging apps.

Re:Pipenv: promises a lot, delivers very little by sametmax on Wednesday December 05, 2018 @07:07PM (#18614088)
> The point of pyproject.toml is to allow other libraries to replace setuptools.

This can be done with setup.cfg. Setuptools is only a backend supporting it. You can create other ones. Poetry and pipenv could support it in a week in their authors decided so.

> The reasons for not using setup.cfg are explained in the PEP.[1]

Those are not reasons, those are excuses. Let me quote it:

>> There are two issues with setup.cfg used by setuptools as a general format. One is that they are .ini files which have issues as mentioned in the configparser discussion above.

Not only setup.cfg does the job with the current limitations of the ini format (while pyproject.toml still doesn't with its fancy one), but python projects are not so complex they require such a rich content.

Besides, nothing prevent PyPA to says that setup.cfg format now has a version header, with the current setup.cfg being headerless version 1, then make the header mandatory for version 2 and increments it to move toward TOML if we ever reach a limitation. That's how formats grow everywhere else in the world.

>> The other is that the schema for that file has never been rigorously defined and thus it's unknown which format would be safe to use going forward without potentially confusing setuptools installations.

That's incredibly dishonest, since I gave a link to a complete documentation of the format in my previous post. Besides, it's better to actually refine the existing standard if you ever find it lacking than recreating one from scratch. While there are good reasons to do so, the later is rarely a rational engineering decision, and most often driven by ego.

>> While keeping with traditional thanks to setup.py, it does not necessarily match what the file may contain in the future

So ? How is that a problem ? A standard is not meant to be set in stone. It evolves. But it can't do so if everytime one has an itch, one reinvents the wheel.

Re:Pipenv: promises a lot, delivers very little by pseudalopex on Wednesday December 05, 2018 @11:11PM (#18615337)
The "existing standard" is whatever configparser accepts. configparser is part of the standard library. Different versions accept different things. The setuptools documentation covers higher-level things like key names. That isn't what they're trying to standardize.

The last sentence you quoted explains why they picked "pyproject" instead of "setup". It isn't why they picked TOML.

Re:Pipenv: promises a lot, delivers very little by sametmax on Thursday December 06, 2018 @03:09AM (#18616249)
See my answer to your other comment.

Also "higher-level things like key names" is half of the standard.

Besides, picking a new (even if better) serialization format is not good reason to create a whole new standard with names, convention, tooling, etc., as explained earlier.

There are sane ways to make the existing system evolves and improves incrementally, using the legacy standards that benefits from the existing situation, and allow the improvements from the new one. All that without the madness of messing with the entire community once again after 15 years of instable package management.

Yeah it's less sexy that having to create your new baby, yes it's less fun than using that new shinny format (and I say that while I __love__ TOML), and less it's less satisfying than having your name as the creator of a whole new jesus-format-saver. But that's the mature and professional things to do.

Re:Pipenv: promises a lot, delivers very little by pseudalopex on Thursday December 06, 2018 @07:13PM (#18623894)
Nothing from setuptools is being standardized. Tools can use any keys they want in any format they want in any file they want.

The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.

The only new tooling for TOML is a small library. A new INI dialect would need one too.

Re:Pipenv: promises a lot, delivers very little by sametmax on Friday December 07, 2018 @03:58AM (#18626089)
> Nothing from setuptools is being standardized. Tools can use any keys they want in any format they want in any file they want.

No, if you use any key, it won't work with setuptools.setup(), and just like a python code that doesn't run with cPython will never be popular, it will not be used.

Also, if you look at how poetry use pyproject.tml, they just create a custom section. So basically, they don't use your standard.

> The "legacy standards" are subtly incompatible INI dialects that people recently started putting into the same files. The incompatibilities mostly don't matter because most tools just read their own sections. They do matter if you want to standardize them.

That's kinda my point for comments and comments. Standardize the status quo, then increment from that. Not sexy. Not pure. Welcome to the real life.

Didn't you learn anything from the distutils/distribute/setuptool mess ? From the Python 2 / Python 3 breakage ?

And could you address any of my concerns instead of just attacking ? Because I'm trying to address yours with solutions. You just write short busts of "no, it's bad, we are good". That's not really giving me trust in your decisions, and it __lowers__ my confidence in pyproject.toml because the people defending it basically are not behaving like engineers trying to solve a problem, but as salesmen trying to only defend their product.

> The only new tooling for TOML is a small library. A new INI dialect would need one too.

But but we can start from a standard that works now, is used already, and is compatible with existing stacks. Instead of arriving with the theorical untested, incompatible best thing that add a layer on top of the mess.

Re:Pipenv: promises a lot, delivers very little by sametmax on Thursday December 06, 2018 @07:12AM (#18617317)
Because I hate the "this ship has sailed" argument about pyproject.toml, here are some github usage stats:

- setup.py: 1,259,007 results (https://github.com/search?q=filename%3Asetup.py)

- setup.cfg: 165,716 results (https://github.com/search?q=filename%3Asetup.cfg)

- pyproject.toml: 2,137 results (https://github.com/search?q=filename%3Apyproject.toml)

Also, remember that setup.cfg is completly compatible with setup.py, the migration is painless. All the legacy tools work. Not the case with pyproject.toml.

Re:Pipenv: promises a lot, delivers very little by Kwpolska on Thursday December 06, 2018 @03:31PM (#18621891)
setup.cfg isn't always used as a setup.py replacement. I use it for configuring flake8, wheel, pytest and coverage: https://github.com/Kwpolska/python-project-template/blob/mas...
Re:Pipenv: promises a lot, delivers very little by IvanAnishchuk on Thursday December 06, 2018 @04:06AM (#18616443)
This is interesting. Somehow I missed this, using setup.cfg make much more sense than putting everything into python code. I wonder if pip-tools will allow compiling install_requires and test_requires in setup.cfg into nice requirements.txt and dev-requirements.txt without too much magic...
Re:Pipenv: promises a lot, delivers very little by guggle on Thursday December 06, 2018 @08:11AM (#18617696)
Your latest blog post say good things about poetry and bad things about pyproject.toml.

However the default in poetry seems to be pyproject.toml... I'm confused.

Re:Pipenv: promises a lot, delivers very little by sametmax on Friday December 07, 2018 @04:03AM (#18626106)
Because I'm not dogmatic, I can regognize good tools, even if I disagree with the underlying political decision.

Computing is not black and white, and perfect purity is only nice in "fizz buzz".

Now to be extra fun, poetry uses a custom section ([tool.poetry]) in pyproject.toml, not really the standard itself. What does that say about this format ?

Re:Pipenv: promises a lot, delivers very little by guggle on Thursday December 06, 2018 @08:14AM (#18617711)
oh, nevermind... I'll stick to manual venv/pip:

  ptest poetry add requests
                                                                                 
[UnicodeDecodeError] 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
Re:Pipenv: promises a lot, delivers very little by sdispater on Thursday December 06, 2018 @09:03AM (#18618084)
Author of Poetry here!

I've never seen that error before.

Which version of Python do you use?

And feel free to create an issue on the issue tracker: https://github.com/sdispater/poetry/issues

Re:Pipenv: promises a lot, delivers very little by sametmax on Friday December 07, 2018 @04:09AM (#18626138)
pyproject debates aside, I love your work. I think poetry is a beautiful piece of software, the source code is very easy to read and the 2/3 compat is well done (although I would not use assert to check things).
Re:Pipenv: promises a lot, delivers very little by agateau on Thursday December 13, 2018 @04:21AM (#18670987)
Looks like a locale problem. `locale-gen en_US.UTF-8` should fix it.