One is that the dependencies are a beast. The risk that it won't find a solution between a number of libraries that are only compatible with certain versions is high.
The other one is that ML projects themselves contain data, often large amounts of data.
For instance a "word2vec" style model might be 1 gigabyte and it might be a part of another model. (Say you turn the words to vectors then put the vectors through a CNN or RNN.) The Python packaging system might be a logically correct place to store this data (a necessary part of the the model) but it's a big file that will cause hassles if you do everything right, big hassles if you do things wrong (like compress anaconda packages with slow bzip2, use whatever algorithm that Docker uses to superamplify I/O, ...)
It is nice to pack all your "empty" models (ready to train) as python packages and maybe even your "trained models" (some data files to supplement the empty files) but it will take some iteration to make it all go smoothly.
Pip’s dependency solving strategy is not correct, especially if you install pip packages one at a time.
A real dependency solver would download the metadata for all matching versions (can be done with two seeks for a wheel but gotta run setup.py for an egg…) do an smt solve and then install the wheels.
Pip just starts installing packages, hopes for the best, sometimes backtracks, often gets stuck or does the wrong thing.