News for hackers. Stuff that matters.
Re:Rye and Uv: August Is Harvest Season for Python Packaging by crabbone on Wednesday August 21, 2024 @04:36PM (#41314008)
> I manage a python monorepo in prod and dependency management is hell

You are either not very competent at what you do, or you wrote the opposite of what you wanted to write.

With monorepo you don't have dependencies. Everything is in the repository. The reason to have monorepo is to avoid dealing with dependencies...

Re:Rye and Uv: August Is Harvest Season for Python Packaging by drawnwren on Wednesday August 21, 2024 @05:06PM (#41314188)
So, you have two choices for directory layout in mono

either:

root/

    requirements.txt / pyproject.toml

    src/ 
---

in which case you need to be careful about when your dependencies get installed during deployment because ie pytorch can be ~500mb

your other option is

root/

    src/ 

        A/ 

              requirements.txt / pyproject.toml

        B/ 

              requirements.txt / pyproject.toml
which means you need to come up with a bespoke dependency management solution. in either case, you're doing dependency management.

I encourage you to google "monorepo {python, poetry, pip}" and you're going to land on many multiple page blog posts describe arcane dependency management solutions.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by gen220 on Wednesday August 21, 2024 @06:51PM (#41314956)
The second option is traditionally what people mean by "monorepo" (multiple executables in a repo with a dependency tree). First one you'd call a "monolith" (single executable in a repo).

The second one you'd want to manage with a tool like Bazel, which will use whatever plugins are appropriate for the language (pip in Python's case).

It's a legitimate ambition to try and build a tool within a single language's ecosystem to manage the second option, but it's a really hard problem that Bazel (and others) have already solved well, so you might as well use them instead.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by drawnwren on Wednesday August 21, 2024 @09:15PM (#41315772)
So you can do a somewhat hacky thing with poetry workspaces to get the first one to contain multiple binaries like i.e. https://github.com/bleu/balpy-poc

But yes, you're right -- Bazel is the most correct monorepo solution. We started with a combination of the second structure, poetry, and shell scripts but have since moved to nix.

I use nix btw.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by gen220 on Thursday August 22, 2024 @09:45AM (#41320208)
Yea nix is an interesting angle of approach. It rhymes a lot with Bazel (reproducibility, explicit/declarative dependency tree, etc.), but nix approaches the problem from a different, lower layer of abstraction (by replacing the OS-level "pip" in apt, pacman, snap, brew, and the like).

I think, in the long run, something like nix will win out. It makes sense for your OS "package build system" to be the same as your project build system.

If your python project depends on some mildly obscure python lib, do you write your own nix "package" (or whatever nix chooses to call them) wrappers for each release you end up using? Any other tripwires in your experience so far?

Re:Rye and Uv: August Is Harvest Season for Python Packaging by drawnwren on Thursday August 22, 2024 @04:51PM (#41324296)
Nix calls them "derivations", but yeah you're right again. Lots of weird language in this space.

Specifically, we use this nix project [1] which provides a nice translation layer between a poetry project and a nix derivation. It allows our devs to use poetry where they want (with local relative paths in the pyproject.toml) and ci/cd to have more granular control of deps at build time.

There are some corner cases. As you correctly guessed, some more obscure python libraries might require a few extra lines (i.e. to specify that this package needs setuptools etc). of code.

1 - https://github.com/nix-community/poetry2nix

Re:Rye and Uv: August Is Harvest Season for Python Packaging by animal_spirits on Wednesday August 21, 2024 @06:08PM (#41314685)
So this example is a single repo with many services? I don’t think OP was talking about that. Why would you have a single repo with different requirements? At that point they should be independent projects
Re:Rye and Uv: August Is Harvest Season for Python Packaging by lijok on Wednesday August 21, 2024 @07:02PM (#41315034)
Because that’s the monorepo pattern. What you’re thinking of is a monolith. Two separate concepts
Re:Rye and Uv: August Is Harvest Season for Python Packaging by selcuka on Wednesday August 21, 2024 @07:05PM (#41315062)
> So this example is a single repo with many services?

Yes, that's the meaning of monorepo. If it was a single service it would be a monolith.

> Why would you have a single repo with different requirements?

Real world example: I implemented a client/server system using Python, and they have completely different requirements (even the Python versions are different). I still want to share code between client and server codebases, so a monorepo is the perfect choice.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by crabbone on Thursday August 22, 2024 @09:29AM (#41320056)
You should never use requirements.txt, nor pyproject.toml and this has nothing to do with whether you put multiple projects into the same repository or not. This is just an all-around bad idea.

But, anyways. In a monorepo project you would have nothing to do with requirements.txt or pyproject.toml because all dependencies are already there. There's no need to install anything from anywhere...

Re:Rye and Uv: August Is Harvest Season for Python Packaging by drawnwren on Thursday August 22, 2024 @04:42PM (#41324232)
How can you not have a pyproject.toml/requirements.txt, nor manage dependencies yourself?

Do you have no external dependencies at all?

Re:Rye and Uv: August Is Harvest Season for Python Packaging by crabbone on Sunday August 25, 2024 @10:47AM (#41347674)
1. In monorepo you don't have dependencies. This is the whole point of having a monorepo: everything is included. You can build on airgapped system, no unexpected inputs into your builds, no network partitioning problems. This is the whole reason why people do that. Total control and stability. But you pay for it by having to maybe manage third-party code yourself. By having to use more space. Probably, you will need more infrastructure to side-step tools that cannot be made to not use network etc. Hope this also answers the question about external dependencies: they become internal dependencies.

2. Why you should never use requirements.txt: the tradition of using this approach comes from total misunderstanding of the goals of project deployment. This is surprising and upsetting, because this isn't a difficult concept. This is due to most developers not wanting to understand how infrastructure of their project works, and the willingness to settle on the first "solution" that "worked". The goal of deploying a project must be that byte-compiled Python code is placed in the platlib directory, accompanying data in the data directory and so on. The reliable way to accomplish this is to make a Python package and install it. Requirements.txt plays no role in this process, since package requirements need to be written into META file in the package info directory. Instead, the process that involves using requirements.txt typically ends up installing "something" that at the time of writing allowed the authors of the project to somehow make their code work, due to the combination of such factors as current working directory, some Python path files placed without their knowledge into platlib etc. This is a very fragile setup and projects designed in this way usually don't survive multiple years w/o updates that keep modifying requirements.txt to chase the latest changes in the ever changing environment.

3. pyproject.toml had more potential, in principle, but turned out to be a disaster. The idea behind this contraption was to organize configurations of multiple Python-related tools under one roof. But the chosen format was way too simplistic to realistically replace the configuration of other tools, and the configuration started to slide down two bad directions: either "string programming" (i.e. a technique where strings in the code develop special meaning and parsing), or delegation (i.e. pyproject.toml would contain a minimal code necessary to redirect to the real configuration). Second problem with pyproject.toml esp. when it comes to dependencies: there was no plan on how to solve this problem. No system. So, every tool that decided to support pyproject.toml decided to do it in a way that suits it. So, in the end, it's always better to just use the native configuration format of the tool that does the package building, instead of involving the middle-man: pyproject.toml. It always stands between the developer and their ability to debug the code, to fine tune the tool they need to run. It's the MS Windows registry all over again, but even more poorly executed.

----

So, what should you do instead? -- Well, there's a problem... there aren't any good Python tools :( And, like I mentioned before, the reason is even not the tools / their authors, it's that the underlying design of Python infrastructure is broken. So, you are bound to choose between the bad and the worse.

On the bright side: the problem, when you really understand what needs to be done is very simple. For any individual project you can solve all your deployment and packaging problems very easily writing in any language that can do infrastructure-related work. I've done this multiple times (as a result of frustration with Python's own tools) and had never a reason to regret it.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by dudus on Wednesday August 21, 2024 @04:48PM (#41314071)
You still have dependencies. But they are included in the repo.

What happens when you have 2 different apps in your monorepo but one uses an older version of Django and wasn't upgraded yet. The monorepo doesn't handle that automatically, you need tooling. It's not as black and white as you say.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by crabbone on Monday August 26, 2024 @08:53AM (#41356664)
In a company I worked for that used monorepo we had multiple versions of Linux (CentOS 6 variants) all at the same time. Trust me, something like different versions of Django is not a really big problem in comparison.

But, to answer your question in a more practical way: what are you going to do with these two versions of Django? Are you planning on running a single pre-fork server with two different Django application servers? Are there going to be two different pre-fork servers? Do both Django versions have to be loaded by the same Python interpreter? Or maybe they don't even need to be deployed on the same compute node?

Once you can answer questions like these, the solution becomes obvious. Most likely, what you want is something like two pre-fork servers proxying HTTP traffic into two separate instances of Django-based Python Web applications. So, in your deployment script, you create eg. two virtual environments and place two different Django packages into these two environments, together with associated code for the Web application.

There are, of course, ways to improve on that. Deploying while using Python packaging is, in general, wasteful. A better way is to merge all the packages you need to deploy your application into a single filesystem snapshot (removing all the info directories, and, potentially, all the source files, replacing them with bytecompiled ones, while also pruning all other irrelevant data from Python packages, s.a. readmes, test files etc.) Going even further, you can Cythonize your code, or even embed Python interpreter into your http server so that you deploy your application as a single binary. And there are plenty more of other options. The sky is the limit really.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by miohtama on Wednesday August 21, 2024 @04:44PM (#41314054)
Generally all Python software use third party packages which are fetched from PyPi package site. These packages are not stored in monorepo.
Re:Rye and Uv: August Is Harvest Season for Python Packaging by TheCleric on Wednesday August 21, 2024 @07:34PM (#41315226)
Not necessarily. Some people do vendor their packages in the repo for consistency (though admittedly it’s rarer).
Re:Rye and Uv: August Is Harvest Season for Python Packaging by crabbone on Monday August 26, 2024 @08:56AM (#41356684)
This is maybe common, but this contradicts the definition of monorepo. You just use the word incorrectly. "Mono" means "one". If you pull packages from elsewhere, that stops being "mono". There's really no difference in this situation between your team publishing multiple packages from multiple repositories and then assembling them together for the purpose of deployment, or doing so, but with the third-party packages.
Re:Rye and Uv: August Is Harvest Season for Python Packaging by dang on Wednesday August 21, 2024 @10:53PM (#41316372)
Personal attacks are not allowed on HN and we ban accounts that cross into that, so please don't do that.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

p.s. Fortunately most of your HN posts appear to be pretty good so this should be easy to fix!

Re:Rye and Uv: August Is Harvest Season for Python Packaging by crabbone on Thursday August 22, 2024 @09:27AM (#41320035)
How is this personal? I have no idea who the person is... I'm commenting on something they wrote, not on something they are.

If a person in a home-cooking forum advises someone to use a butter knife to slice tomatoes, it's highly appropriate to tell them they are incompetent and shouldn't be advising anyone. This post is exactly the same: an absurd advise that can only be explained by either an honest mistake or incompetence.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by pvg on Thursday August 22, 2024 @10:37AM (#41320769)
it's highly appropriate to tell them they are incompetent

It's not on HN because it's a personal swipe and the HN guidelines ask you not to do those. It might work on your cooking forum but we know, empirically, that it doesn't work well here.

Re:Rye and Uv: August Is Harvest Season for Python Packaging by kstrauser on Thursday August 22, 2024 @10:15AM (#41320549)
They didn’t say they were doing something goofy like that. They said they were running into problems at work. It could well be that they have a million-LOC repo that’s enormously complex with decades of legacy setup to consider. Presume that they’re competent at their job, and maybe what they’re trying to do isn’t trivially easy.