News for hackers. Stuff that matters.
Re:Ask HN: Why Poetry did not become a mainstream package ma... by PaulHoule on Sunday June 20, 2021 @06:55AM (#27568879)
If you think ‘pip works’ then the programs you are building are too simple. I worked at a place where we did machine learning builds and neither pip or conda worked reliably. What was insidious was that it almost worked and people thought they could live around it’s limitations which turns every 2 hour job into a 15 hour job. (But they are data scientists…. A reliable build is like Bigfoot, cold fusion or Linear A to them.)

Pip’s dependency solving strategy is not correct, especially if you install pip packages one at a time.

A real dependency solver would download the metadata for all matching versions (can be done with two seeks for a wheel but gotta run setup.py for an egg…) do an smt solve and then install the wheels.

Pip just starts installing packages, hopes for the best, sometimes backtracks, often gets stuck or does the wrong thing.

Re:Ask HN: Why Poetry did not become a mainstream package ma... by rocmcd on Sunday June 20, 2021 @09:39AM (#27569841)
That's a fair point, most of the applications I've worked on have been relatively simple (or I've just gotten lucky). It makes sense that different issues crop up at different scales, given the complexity of different workflows and package dependencies. I also haven't worked in ML before, so that may be a class of complexity all on its own.
Re:Ask HN: Why Poetry did not become a mainstream package ma... by PaulHoule on Sunday June 20, 2021 @08:13PM (#27574994)
Two issues turn up in in ML.

One is that the dependencies are a beast. The risk that it won't find a solution between a number of libraries that are only compatible with certain versions is high.

The other one is that ML projects themselves contain data, often large amounts of data.

For instance a "word2vec" style model might be 1 gigabyte and it might be a part of another model. (Say you turn the words to vectors then put the vectors through a CNN or RNN.) The Python packaging system might be a logically correct place to store this data (a necessary part of the the model) but it's a big file that will cause hassles if you do everything right, big hassles if you do things wrong (like compress anaconda packages with slow bzip2, use whatever algorithm that Docker uses to superamplify I/O, ...)

It is nice to pack all your "empty" models (ready to train) as python packages and maybe even your "trained models" (some data files to supplement the empty files) but it will take some iteration to make it all go smoothly.