I'm the CTO at a data science company, and this has been my experience too. I've been lucky enough to have quite a few engineers go from zero practical experience to being able to train and deploy complex ml solutions, and the most successful solutions have always involved a combination of just a couple of tools:
- airflow and/or celery for running data extraction and transformation jobs
- pandas and numpy for data wrangling
- sklearn, xgboost, lightgbm, pytorch or tensorflow for training/inference
- flask or Django to serve results
It's a handful of technologies, but they're (generally) mature, battle tested, and well documented.
Re:We Need DevOps for ML Data
by softwaredoug on Tuesday April 28, 2020 @08:32PM
(#23013884)
Generally true. Though I will say that in larger orgs, you will occasionally get someone doing some ML they read a paper on that's not well supported by major tooling. I mean it's the same trend chasing you see in engineering...
It's easy to do "dev ops" for machine learning. Basically, just automate everything and implement gatekeeping mechanisms along with active monitoring.
It's true, though. I had to cobble together a lot of custom things at the time, but it wasn't that hard to do.