News for hackers. Stuff that matters.
Re:Python Wheels Crosses 90% by m4rtink on Monday June 08, 2020 @05:49AM (#23455141)
Pinning dependencies to specific version is also bad - that's lazy engineering & instant technical debt.

You really should check what APIs your dependencies provide and how stable that API is, then set minimal dependency version. If you set the dependency to a specific version, you are creating a headache for anyone trying to use your software later on (by potentially forcing them to use oudated and insecure software) and for distro maintainers trying to keep distro software up to date and secure.

What if the dependency is so unstable that you have to pin the version or even use custom patched version? Well, thenaybe its not somethong you should be depending on & rather use something with less features but more stable. Othervise you are really being unresponsible - using quick convenience at the cost of long term usability of your software.

Re:Python Wheels Crosses 90% by dgoldstein0 on Monday June 08, 2020 @07:06AM (#23455513)
I'm not sure exactly what form of dependency pinning you are hating on - but if you are suggesting we should ditch dep-pinning altogether, I have to totally disagree. It's never a good experience to wake up one day and see that your code that worked yesterday, no longer does; or worse, that it works for you but other people start saying otherwise. Failing to pin deps (e.g. with a lockfile) makes it rather easy for these bugs to slip in - it's just one accidental backwards-incompatible change in a transitive dependency of yours away, and good luck figuring out how to reproduce the problem reliably.
Re:Python Wheels Crosses 90% by m4rtink on Monday June 08, 2020 @07:58AM (#23455755)
I should have mentioned that I'm seeing this from the point of view of a distro maintainer involved in Fedora.

The thing is that in general you don't want to have many version of libraries available in the distro in parallel as each has to be separately maintained, security patches applied, build issues fixed, etc., eating valuable maintainer time. Also in some cases libraries of different version can't coexist on a system cleanly.

Then imagine every piece of software just pins versions of their dependencies to a specific random version that happened to work at the time. To satisfy those arbitrary dependencies the distro would have to maintain all these versions at the same time, which is simply impossible resource vise, not to mention incredibly wasteful (both in maintainer time & system resources).

As for stuff breaking if you don't pin dependency version - well, distros have mechanisms to handle that. For example for Fedora, there is a stable release very 6 months & stable releases are not expected to get major changes in libraries, just bug fixes and smaller enhancements.

And at the same time there is a rolling version of Fedora called Rawhide, where all the latest package versions land and where integration issues are addressed. So any breakage would happen on Rawhide and be addressed by maintainers (of the library/software affected or both) long before a new stable release is cut from Rawhide and users will actually use it.

For an example I'm maintaining the PyOtherSide Qt 5/Python bindings on Fedora. A while ago the build failed due to Python being updated to 3.9. I reported the issue upstream, which quickly fixed it and I've built an updated version in Rawhide. All this long before a stable Fedora version will get Python 3.9, but I can be sure that when this happens, all will work fine.

Re:Python Wheels Crosses 90% by pjc50 on Monday June 08, 2020 @10:10AM (#23456595)
Yes - although from the developer perspective, they want to work cross-distro without having to test a big compatibility matrix, and generally use the language's package manager (which hopefully behaves the same across all platforms) rather than the distro's one.

Especially with Python, and especially with the 2/3 split, people got used to assuming that the distro version was something broken to work around (e.g. Redhat, OSX), and that all "real" work happened in one's local language-specific package cache or venv.

I'm increasingly of the opinion that it's a mistake for distros to ship Python or Ruby packages in their distro-specific package format, but I can also see that's going to be a holy war.

Re:Python Wheels Crosses 90% by dgoldstein0 on Monday June 15, 2020 @02:11AM (#23524295)
the compatibility matrix is exactly what I'm wondering about too. It's far easier to test against a single version of a dependency and deploy against it too.

I can't help but wonder if there's a better way that could scale to open source. I personally like monorepo-based development a lot - where you have one version of every library for the whole repo, and a total ordering on changesets (and a strong test suite to catch regressions). But organizing open source into a monorepo or even a virtual-monorepo seems tricky. Might be the sort of practice that can work well for companies but not so much when decision power is more distributed.

Re:Python Wheels Crosses 90% by sergeykish on Monday June 08, 2020 @12:04PM (#23457778)
As developer I use language specific packager (bundler/gems) and I know it well.

As Linux user who sometimes build packages (tinker around) I need all kind of dependencies - python, ruby, perl, haskell. It is much safer and faster to use distro packages.

Breaking changes on major version is awful for any consumer. Python 2/3 story is a shame. These can't be arguments against distro packages.

Re:Python Wheels Crosses 90% by AnIdiotOnTheNet on Monday June 08, 2020 @09:32AM (#23456325)
So if I understand you correctly, you're asking developers to change how they work in order to make package maintainers' lives easier while providing no benefit to either developer or end user. Is that correct?
Re:Python Wheels Crosses 90% by m4rtink on Monday June 08, 2020 @11:04AM (#23457102)
The benefit for developer is maintainers report compatibility issues before the developer hits them and in general take over a lot of the user support as well as the menial but necessary packaging tasks. For the user the benefit is they can actzally deploy the thing in a real production system (that is not the developrs laptop) with potentially clashing libraries and be reasonably sure it continues to be secure in the future by using up to date maintainer libraries, not some unstable release no longer getting any security fixes the developper pinned it to 5 years ago and never bothered to change.
Re:Python Wheels Crosses 90% by AnIdiotOnTheNet on Monday June 08, 2020 @01:59PM (#23458954)
On the other hand, inserting a volunteer middleman between the developer and the user cause extra unnecessary friction, delays, and has even been the cause of bugs [0], not to mention sometimes they explicitly ignore the wishes of the original developer.

[0] Remember when Debian generated predictable random numbers because a maintainer wanted valgrind to shut up?

Re:Python Wheels Crosses 90% by m4rtink on Monday June 08, 2020 @07:46PM (#23462075)
On the other hand, stuff like the NPM leftpad issue can't easily happen as the package maintainer "middle man" usually inspects any upstream updates for general sanity. And even if it got in to Fedora for example, it would most likely break only the integration rolling release called Rawhide, where it would be quickly discovered and fixed, never reaching the overwhelming majority of users that are on Fedora stable releases.

Now in comparison people using NPM just blindly pulled random stuff directly from upstream without anyone doing any sanity checking at all - no wonder one package vanishing made the whole thing fall over, often directly in production.

Re:Python Wheels Crosses 90% by AnIdiotOnTheNet on Tuesday June 09, 2020 @09:28AM (#23466475)
Are we talking about source-code packaging for developers or binary packaging for users? Because the left-pad scenario really doesn't apply in the latter case.
Re:Python Wheels Crosses 90% by ClumsyPilot on Monday June 08, 2020 @10:58AM (#23457043)
By that logic 90% of javascript ecosystem should never be used in production. Maybe that's even the correct conclusion?
Re:Python Wheels Crosses 90% by m4rtink on Monday June 08, 2020 @02:11PM (#23459070)
I think that's the correct conclusion. ;-)