Interesting, hah - I'm of the opinion that even just the first 3 steps you listed, before you actually get to the learning part itself, can each contain a world of hidden assumptions that can get you mired in a cycle of constantly repairing and tweaking pipelines

You're totally right. All of the ML hype out there isn't helpful either.

The hardest part for me is not the tooling or processes, but ironing out expectations, assumptions, what can and can't be done (or what's worthwhile) with ML.

Getting to a common understanding about metrics and outcomes with an end user / customer can chunks of time out of a project.

The problem is simple, except that Data Scientists seem to resist good software engineering practices and storm ahead to the model development part which leaves a lot of mess for engineers to pick up. MLOps then looks like a 'difficult' problem because of this mess.

That's quite a simplified and antagonistic comment.

Data scientist have their own problems too, like crappy data (if any) and unclear business objectives. This usually means fast iteration at all costs. That's why Python or R are the languages of choice, which are not precisely the most amenable to deployment. Now, think about their KPIs vs yours and everything falls into place.

I view the job of the MLOps framework as essentially a contract where "You place your messy code that does X here, and we'll take care of turning it into a retrainable, resilient, secure API endpoint without any extra fuss". This requires the software engineer/dev ops person to think through ahead of time what the data scientist will need to do as a part of their process and templatize it and form a good abstraction. The problem is that the abstractions are often leaky so there is some constraint that needs to be put on the data scientist. This also requires the MLOps engineer to work on new feature development to form a new abstraction to handle the new features. "Oh you want to do active learning? What does that look like?".

Agree that those steps capture the process.

But whose job is it to run those steps? And note that most of what I think of as DevOps might have an analogy, but no direct equivalent.

Of course this is all solvable; the thing is that it is new. DevOps assumes that changes are to the code, and has processes and tools for managing code changes.

In ML, it is the DATA which changes.

(I'm biased, as I'm the CTO/co-founder of Efemarai)

Yes, there are usually several changes that need to be tracked - code/model changes (those usually happen early on and the stabalise), input/code changes (e.g. pre-processing the data with either new transformations or _other_ models), and data changes (both changes for training and testing). At Efemarai we are thinking about it as any changes to the above should automatically trigger a test suit of the model/process. And under test we're thinking not just the different forms of unit testing the input/output formats and sizes from the model, but also unit testing on the model performance of the data you've collected + stress testing the model with data the model is expected to see in production.

So in reality, it's indeed nothing new, but the standard DevOps pipeline needs to be extended to work with the ML assumptions.