- People thinking if Jupyter notebooks had just the right CSS and animations, it'd make machine learning unstuck.
- People coming from web dev and thinking that ML folks are somehow unaware of Docker/CICD/Kubernetes/Git and if they could just be enlightened, they'd be much more productive.
- People coming from a background of publishing blog posts/Youtube videos about machine learning, or who've worked with cute datasets who think that the actual workflow is someone giving you a CSV and a metric to optimize for à la Kaggle and building something around that workflow. This is for ML what Aikido is for fighting: it only works if your opponent knows the script and agrees to fall when you go Hayaaa!
Disclosure: we're working on our machine learning platform[0] because we desperately need it. We do not come from a web dev or blogging background, and it shows. We're building it because we spent seven years actually delivering machine learning products to paying enterprise in the mid six figures contracts with actual money, trust, and careers on the line. We've done it OK enough to have companies trust us and become repeat customers, but, anyone we faced the problems that anyone who has worked on actual projects has faced. Just getting to formulate the problem is akin to extracting Ramses II's brain through his nostrils. Agreeing on the data to use and then getting the data to a usable form takes a long time. Defining the real world metric is not obvious [if only it just boiled down to F1/AUC/ROC/ETC]. Most things that matter are done by a team, and tools work great for one individual, but break for teams. Just getting results to the client makes the work an endangered specie in an "only one result out of 100 makes it to the client".
It is unbelievable how much these problems impact the team. People can quit over this and we've suffered a lot.
So we're building around our workflow that has generated actual revenues with very large organizations that were satisfied enough to come back for more projects. It is not a perfect workflow, but it is from the real world on real projects and real clients that paid real money to solve real problems.
And we're building it for ourselves to use on actual projects. We aim for 10x improvement.
[0]: https://iko.ai
What we have here is the: "I don't get it, why don't you read this 500 pages manual and type in those 10 commands to print your document" era of computing. People wondered why people had trouble "just" programming something.
That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X.
If data scientists just wrote better code. If developers just were more familiar with machine learning. If ML engineers just could be more CICD aware. If domain experts just could get it right. This is not working.
The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data.
Once we take the collaborative aspect out, and we see data analysts who never had to check out someone else's work to apply minor changes or have someone else apply minor changes to their own code, then we get to the same naive criticism directed at basic organizational practices and tools as we see here.
In your experience, what do you attribute this to?
One way to look at it is that programmers tend to see notebooks as code, and think it makes sense to use the tooling used for code.
Another way to look at it is as a document, and build around that.
Would fiction writers benefit from learning Git ? Sure. Could it be more productive to reduce the activation energy to accomplish the job of tracking changes with another implementation ? I think yes. There are more writers who use Google Docs than there are who use Git.
For example, we added collaborative editing on notebooks[0]. Multiple people can work simultaneously on the same notebook, see each other cursors and changes live. It's a much better experience. We also added multiple checkpoints behind the scenes[1].
I believe that as programmers, we have the opportunity to create better abstractions around workflows to simplify what we take for granted and do as second nature for others.
One issue is cognitive load. People having to remember to do things. We strive to make it automatic. One example is for experiment tracking: sure, they 'just' have to track experiments and add the necessary code, but we do it automatically so it happens without them remembering. Models are detected are saved, parameters and metrics are logged, etc. This considerably reduces the barrier to entry as even a beginner who knows nothing about tracking gets their experiments tracked.
This is what we're doing. Handle complexity on behalf of a user, and reduce the skills required to get things right. Same for deployment. Can they learn Docker? Sure. Why do they have to, though. So we automatically build model images for them, and they can deploy them themselves clicking a button. As I said, these are problems we've had in the past, for example a 'data scientist' having to sollicit the help of à colleague to deploy their model. This was a bottleneck for us.
Choice excerpts:
* co-ordinating code between team members by having everyone periodically copy their code to one team members personal google drive, where one of them would then spend a day fixing merge conflicts by hand. When asked why they were doing it like that, would they like any help, git can probably solve these problems, they asserted that this worked fine with them.
* Spinning up machines in EC2, then keeping track of keys (new one for every machine obviously) by storing the Pem files in an S3 bucket when they remembered. This didn't happen, so instead we had a dozen machines nobody could get into anymore after they left...
* Calling Python code in another file, by running os.spawn, calling a python interpreter with the given file, passing in parameters a cmd line arguments, and getting the result by waiting for a specifically named file to be written to disk. When asked "why are you doing this?" I was told "because it was easier".
* All the python they wrote eschewed functions were possible, with everything at root-indentation & no import guards (so it couldn't be imported safely). Functions, when they were used relied heavily on global variables and other objects/imports/etc being present implicitly. There were also logging and json serialisation functions that were hand written. When I asked why this teammate did this they replied "it only took me a couple of minutes and it works? What's wrong?" Unsurprisingly, it didn't serialised datetime objects correctly...
* Re-inventing the functionality offered by Pythons multiprocessing/ Joblib libraries-but worse- by shelling out (again) and manually calling a shell script to detect the number of CPU's and then using x-args to spawn the python processes. "Well we've written it now and it works but maybe next time". Unsurprisingly, these processes would fail, silently and lose data.
* Using eval() in production code because they "couldn't figure out a better way of getting the value of the attribute" despite being told that getattr is safer and faster.
* Maintaining a spark cluster full of Jupiter notebooks unhelpfully named things like STEP_301.ipynb, STEP_405.ipynb and not anything that told you _what_ they did. Figuring out which notebooks called which other notebooks and where any bit of code came from was, unsurprisingly literally kafkaesque. These eventually got checked into git (root credentials, embedded images and all), in a single commit with the message "add to repo".
* Instead of reading the documentation, or using something like SqlAlchemy, code samples of how to connect to various different types of database (mysql, Postgres etc) were stored in a random google doc (once again, one of their personal accounts) and access was passed around.
GUI tools only help with that to some extent, too, and only the right ones.
Although I really dislike GUIs in general so maybe that's it.
It has more users, I think, than even the “visual” source control systems that are supposed to be more usable.
Git is widest adopted, period (EDIT: because of how it blossomed out of linux dev, essentially). I was able to stick to bzr or fossil for a lot of things until github and gitlab basically became where open source work (and even corporate transitioned to some extent!) needed to happen, and the friction of using anything else is just too great at this point in time.
Which is a shame, really, as --fast-import/export is actually not a bad way to transition from one scm to another these days, and I'd personally welcome a little more active competition in the space to evolve beyond the current status quo.
In your personal opinion, what's so "bottom up" about checking out code from a repository and committing changes to a branch?
> (...) while at the same time it exposes an interface that makes it all to easy to screw things up catastrophically.
What? Care to point out a single example of what you perceive as being one of those "catastrophical" failures?
In all my professional life, which includes a bunch of years as a data analysis, I never experienced a single failure with git. Not a single one.
In fact, the only failure I ever experienced with git came from a stupid error I once made by tracking a git repository which found its way into a Dropbox shared folder.
Care to shed some light on what you perceive as being your pain points?
Fundamentally it is a same as git: you can push, pull, you have commits, branches, merges, etc... and branching is easy.
It is mostly a difference in philosophy. While git will leave you with lots of powerful and flexible tools, Mercurial will be much more limited out of the box, advanced features are extensions you have to enable.
The Mercurial history is also considered sacred, and everything is done do discourage you from modifying it, commands like squash and rebase are hidden behind extensions, branches are also permanently attached to commits (the equivalent of git branches exist and are called bookmarks).
There is also no staging area. Instead of "remote" branches, there is a simpler "phase" system that tell you which commits are local and which one have been pushed.
All that makes Mercurial easier to use, especially for non-developers. And in fact, most people who tried both preferred it over git, with the exception of git experts, who prefer git for its speed and flexibility.
tbh I'm mostly happy with git. It does what I need it to do. Yes, it could be easier and more transparent, but everything is a trade-off.
People I work with don't care that our notebook tools have rudimentary syntax highlighting or type-checking. We don't rely on our tools to make sure our code is written correctly. We just memorize the APIs and rely on experimentation to get things working.
Please don't make us use git. Checking out and checking in files is too slow! Oftentimes, Slacking code to a colleague is the best way to collaborate, and the fastest.
Don't make us use confusing and broken authentication schemes to access our data! I work best if I can just download live data to my machine and work locally. If it's too big or it's something like streaming, just give me an admin password that I can save in my notebook and use to hit some database clusters. Nothing wrong with that!