Ask HN: Machine Learning Engineers, what do you do at work?

Re:Ask HN: Machine Learning Engineers, what do you do at work? by fragmede on Saturday June 08, 2024 @06:29PM (#40620619)

Disk space is cheap, so where it's possible to have 2 (or more) venvs, that seems easiest. The problem with venv is that they don't automatically activate. I've been using a very simple wrapper around python to automatically activate venvs so I can just cd into the directory and do python foo.py and have it use the local venv.

I threw it online at https://github.com/fragmede/python-wool/

[ Parent | on HN ]

Re:Ask HN: Machine Learning Engineers, what do you do at work? by sjducb on Sunday June 09, 2024 @03:23AM (#40622639)

You’re already managing a few hundred dependencies and their versions. Each venv roughly doubles the number of dependencies and they all have slightly different versions.

Now your 15 venvs deep, and have over 3000 different package version combinations installed. Your job is to upgrade them right now because of a level 8 CVE

[ Parent | on HN ]

Re:Ask HN: Machine Learning Engineers, what do you do at work? by fragmede on Sunday June 09, 2024 @02:27PM (#40626428)

Yeah that sucks. something like:

    for venv in $(find ~/projects/ -type d -name 'venv'); do
      (
        source ${venv}/bin/activate
        pip install --upgrade pip
        pip install --upgrade package_with_cve
        deactivate
      ) &
    done

should do the trick. Sucks that we're in that world, but let's not work any harder than we have to.

[ Parent | on HN ]

Re:Ask HN: Machine Learning Engineers, what do you do at work? by fragmede on Monday June 10, 2024 @03:54AM (#40631228)

I forgot about parallel, which would do better than spawning off all the pip install --upgrade at once.

[ Parent | on HN ]

Re:Ask HN: Machine Learning Engineers, what do you do at work? by tmnvix on Sunday June 09, 2024 @04:52PM (#40627527)

After many years of Django development I've settled on what I find the simplest solution. It includes activation of virtual environments when I cd into a directory.

pyenv for installing and managing python versions.

direnv for managing environments and environment variables (a highly underrated package imo).

With those two installed I just include a .envrc file in every project. It looks like this:

layout python ~/.pyenv/versions/3.11.0/bin/python3

export VARIABLE1=variable1

export VARIABLE2=variable2

etc...

[ Parent | on HN ]

Re:Ask HN: Machine Learning Engineers, what do you do at work? by fragmede on Sunday June 09, 2024 @11:59PM (#40630013)

Oh nice. Stick

    PYTHONPATH="`pwd`/venv/lib/python-3.11/site-packages"

or whatever in .envrc and Bob's your uncle.

[ Parent | on HN ]