News for hackers. Stuff that matters.
Re:Log4j: The pain just keeps going by alfalfasprout on Wednesday July 20, 2022 @12:59PM (#32168689)
We actually are running into this exact problem with running production ML workloads and despite anyone's claims it's not a solved problem. In fact, this log4j mess is a cake walk in comparison.

The issue: ML workloads are very sensitive to the exact version of a framework/library used and these frameworks/libraries are not stable despite what their semantic version says.

So what do you do when you have a python vulnerability? Best case, an update "just works" but in many cases you either need to retrain the model (can be extremely expensive or inconvenient), rewrite it if APIs changed, or hunt down a difference in model output if some downstream dependency broke something. Stability of software in the python world is abysmal.

Re:Log4j: The pain just keeps going by xmprt on Wednesday July 20, 2022 @01:12PM (#32168909)
Is this an issue of code stability or do these libraries have leaky abstractions that cause issues with models trained on previous version, because the library implementation changed and since ML has low explainability, it's hard to tell how the change impacts the model performance? eg. making a library run faster or use a slightly different parameter internally causes some floating point imprecision that cascades into wildly different results?
Re:Log4j: The pain just keeps going by alfalfasprout on Wednesday July 20, 2022 @01:18PM (#32168996)
Unfortunately it's all of the above. Sometimes the code isn't stable (eg; a library method changed), sometimes the behavior has changed (eg; same library method but now it has a different behavior), and sometimes it has some other unwanted effect (the computation produces a totally different result, it runs way slower, or something else).

With a service it's all much easier because you can have pretty robust integration tests and you're good to go. With an ML model it's a lot trickier because even if you have sample payloads and responses there might be other payloads you didn't think about that could cause issues. Or it might rear its ugly head when you finally retrain the model and suddenly model performance is crap.

Re:Log4j: The pain just keeps going by dotnet00 on Wednesday July 20, 2022 @02:09PM (#32169848)
There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This then breaks models when the framework updates.

One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues.

I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.

Re:Log4j: The pain just keeps going by aclatuts on Wednesday July 20, 2022 @02:02PM (#32169751)
I find Python is very prone to a lot of breaking changes even with minor version changes. And not just that, even minor versions have to match up specifically with another libraries minor version. It seems to be a community or ecosystem problem with Python, which is why I avoid Python as much as possible.

In every other ecosystem when minor version are updated there are very rarely issues, and there are little issues with compatibility with minor versions of libraries.

Re:Log4j: The pain just keeps going by btilly on Wednesday July 20, 2022 @02:45PM (#32170377)
My (somewhat dated) experience says that Ruby is all the way worse than Python on this.

In part due to a culture that encouraged monkeypatching. With the result that what behavior you get depends on who last monkeypatched it. So a dependency 3 deep that loads the module you expected to load last suddenly causes the monkeypatch you were depending on to disappear...

Re:Log4j: The pain just keeps going by treeman79 on Wednesday July 20, 2022 @04:46PM (#32172084)
Monkey patching was the greatest thing ever. Until the community realized how bad it was. Don’t see it much these days
Re:Log4j: The pain just keeps going by skrtskrt on Wednesday July 20, 2022 @02:43PM (#32170340)
eh, people love to break stuff on minor version changes of Go libraries to
Re:Log4j: The pain just keeps going by wizofaus on Wednesday July 20, 2022 @03:34PM (#32171010)
At least you find out at compile time...
Re:Log4j: The pain just keeps going by kag0 on Sunday July 24, 2022 @01:33PM (#32215699)
is this ML workload an "I trained a model, pickled it, and now we ship that around in production" type situation?
Re:Log4j: The pain just keeps going by einpoklum on Wednesday July 20, 2022 @03:12PM (#32170737)
> So what do you do when you have a python vulnerability?

If you are willing to run other people's arbitrary (Python) code, then - vulnerability or no, you should consider the system running this code as compromised.