Re:Why software engineering processes and tools don’t work f...
by kelnos on Thursday October 15, 2020 @01:36AM
(#24785233)
Most of the data science / ML code I've seen reminds me of academic researcher code. Hack and experiment until it kinda works, never write tests because what's the point, it works on my machine and that's good enough.
For a ML-heavy project I ended up being the "productionize this stuff" guy. My favorite was a bunch of python code that depended on a library that under the hood would spin up a new JVM instance to do some NLP work every time it was called (or whatever incorrect way they were using it caused this to happen). And they wanted to do this in the context of a synchronous HTTP request/response cycle.
My overall impression was that these were incredibly smart people, but they had no interest in understanding the context in which their code was going to run, or in understanding business requirements around reliability or maintainability. They just wanted to hack on cool projects that they found interesting.
Given that, it doesn't really surprise me that a lot of ML/data science people don't "get" accepted software development processes. There were some who made the leap and learned to understand why some of these practices are useful and tend to give you a better product, but they seem to unfortunately be in the minority. This is a pretty young field/subfield, and I think it's to be expected that practices are pretty raw and underdeveloped.
For a ML-heavy project I ended up being the "productionize this stuff" guy. My favorite was a bunch of python code that depended on a library that under the hood would spin up a new JVM instance to do some NLP work every time it was called (or whatever incorrect way they were using it caused this to happen). And they wanted to do this in the context of a synchronous HTTP request/response cycle.
My overall impression was that these were incredibly smart people, but they had no interest in understanding the context in which their code was going to run, or in understanding business requirements around reliability or maintainability. They just wanted to hack on cool projects that they found interesting.
Given that, it doesn't really surprise me that a lot of ML/data science people don't "get" accepted software development processes. There were some who made the leap and learned to understand why some of these practices are useful and tend to give you a better product, but they seem to unfortunately be in the minority. This is a pretty young field/subfield, and I think it's to be expected that practices are pretty raw and underdeveloped.