> You can go about doing more math, I'll do the engineering

I'd say most of the teams I supported as a backend/DevOps/infrastructure engineer followed that pattern.

Generally the evolution I saw was start with a handful of sciency folks working in R or python, get more grant money as work becomes more important, have some SE's thrown in the mix to re-write in python and/or C/Cpp.

A lot of those SE's would float between teams with one of us infra folks to performance tune as they scaled.

> the data scientists are never on-call,

Yeah, but I'll keep my on-call rotation over their constant 11th hour shenanigans trying to get a paper out the door.

I mean, this is probably how things should be done.

Having the specialists of each field appropriately resourced, focused and aligned streamlines everyone’s life. The problem comes in that most places either get the scientists to pull double duty and we end up in the scenario scientists are pushing messy, subpar code out the door that nobody else wants to touch because they’re unsupported and operating out of their depth, and devs don’t want to touch it because they’re not keen to fix someone else’s hacky code and because they probably have other priorities and the science stuff likely isn’t well integrated into the rest of the other code-delivery and monitoring systems.

The big thing is prestige, it takes some pretty big brains to write good, clean production infrastructure that doesn't fall over every five minutes.

Those big brains are absolutely capable of learning the math and doing the creative work of modeling.

They want to do the creative work.

Which means that if you create a prestige gradient and don't let your engineers do interesting things they themselves might get a paper out of, you lose good engineers and get bad engineers instead.

This is also a constant problem. In DevOps and SRE, the very best devops and sre people are absolutely incredibly mercurial and mercenary - because they've been fed total lies about building systems again and again when the net job description is "hey ops person, this application I wrote is misbehaving can you do advanced troubleshooting on this server for me do I can go do leetcode".

This. There is no clean application of comparative advantage here. The great individual that will rewrite the code and knows all the details to get the model scalable, robust and production ready, can also do the data science "creative" work.

If you are not willing to grow to become that person you are only damaging yourself long term. And if you don't attribute that individual's work: a) this is unethical b) they will, and should leave.

P.S. Seen this firsthand and second hand.

Good software dev should be able to refactor the subpar code

Sure but it's less fun to be a refactor assistant full time

> often needs to be optimized for scale or refactored for reusability

It's so easy to make subtle assumptions when you redo their code that completely invalidates the work they've done. ML completely collapses on extremely small errors. Handing something off to someone else to refactor is a dangerous step in the process that risks everyone wasting their time

> It's so easy to make subtle assumptions when you redo their code that completely invalidates the work they've done.

It's equally easy to test whether those assumptions actually break anything. Especially when minor errors can be catastrophic.

I'm part of a team that was tasked with producing a web app from something that was originally a piece of Matlab code.

We considered just running the Matlab code in a container but ultimately IEEE 754 is IEEE 754 regardless of platform/language, so creating a 1:1 implementation in C++ proved possible.

The catastrophic errors aren't the problem, the problem is when things are degraded by 20% which results in a small loss when it could have been a big win. You just assume it didn't work when in reality it wasn't implemented correctly

Personally, I'd like to do both the data science and the engineering. I'd also like to be in a team where each member does both. I at least "feel" like these two skills are close enough spiritually to one another that you can find a team composed entirely of people that do both.

> I think that comparative advantage here is a real and important

It is.

But so are transaction costs, and all handoffs have them.

Interestingly, I'm a physical scientist in a business that makes hardware and software. I'm up front with people, that I'm not an engineer, and nobody expects me to be one. I write code to support prototyping and testing, but don't expect it to go into production. Nobody wants my janky prototype. I produce a theory of operation that covers a proposed design, as well as an outline of the basic manufacturing and service processes needed to make it work. The prototype sometimes helps confirm that the technical requirements for the product can be satisfied at an early stage of a project.

When I do share code and things that I've designed, it's usually for tooling not product, e.g., a script that helps calculate non-obvious design parameters and tolerances. Often, my code is used to test hardware components before the official software is ready.

In my case, I'm on call, though not 24/7, because the business as a whole isn't. For instance I'm available to diagnose supplier and production problems, and deal with weird issues that emerge in the field.

Why do you want them to write code at all, then? Why not just task them with writing user stories around the parameterized functions they need, and just let you figure out how to implement it all?

You usually have to evaluate the model as you develop it, it's not something that a story driven model will fit

The idea that creating a model can be a “user story written in math” seems to me a variation on a common misunderstanding about what model creation, particularly the role of coding in it is. Data scientists, statisticians, modellers, don’t go in knowing what the model is and just coding or specifying it. They use code as a exploratory tool to test several hypotheses until they find one that seems to hold - the code, the data, the algorithm, the statistical test, the exploration and the reasoning are all tools in the process of discovering the right model. It simply can’t be specified in advance, it has to be run and tested.

Having som spaghetti code is a natural consequence of this exploratory, iterative process. Applying good SW engineering practices to this exploratory endeavour is just a natural consequence of the process when you don’t know if your code will be of any use before you finish running and checking the test results. Why would you bother modularising and doing test coverage on something that is very likely to be thrown away after one or 2 runs?

I say this from 28 years of experience both as a data engineer and data scientist. I am a good python developer, and I can write production grade code. But I won’t refactor my code into that until I know that is the code that generates the right model. And I certainly can’t specify this particular code before writing some dirty version of it, testing it and confirming that the model it trains passes some statistical tests, at least.

Basically, the code is not the product - the product is the result of applying some transformations on data and running that through some ML or statistical algorithm to generate a model. Transformations and algorithm being unknown to be useful until tested, hence specification being unknown until coded, run and tested.

Assuming you're writing actually detailed and complete user stories and those user stories are for functionality that basically amounts to "just math" (i.e. no UI or API standards or whatever to worry about alongside the business logic) I imagine it's not much more difficult or time consuming to just write the code rather than describing it in a user story.

Having one person write user stories and a second person, who understands the domain less than the first person write the actual code would be twice as much work for a worse result.

What's the best way to write those user stories? Certainly not plain English; you'd probably want to use some kind of pseudocode. Ah, but it would be good if you could run that pseudocode and see what the results are in the happy path, without worrying too much about what happens when something goes wrong. We have that, it's called Python, so that tends to be how these user stories are expressed.

I expect that it makes sense to have the modelers do at least enough coding to test the common use cases/expected inputs for the model to validate it before handing it off to be "properly" implemented.

Right but they still need to know some code and talk programming language.

The "here is some shitty code that does the job, make it good and production-ready" requires far less communication and domain knowledge than them telling you how exactly it should work and you translating that into code

That’s an unusual style of operation. It may be a really good one, but we don’t really see a lot of teams where people have complimentary skills although really we should.