From physics I have a good theoretical grounding in how ML works (optimizing a cost function over a high dimensional manifold to reconstruct a probability distribution, then using the distribution for some task) but I personally find actually ‘doing ML’ to be rather dull.
I can relate to this a lot. In my company many things you can sell as "AI" can really be solved with traditional data processing.
Fad-chasing often leads to silly technical decisions. Same thing happened with blockchains when they were at the peak of the famous hype cycle. [0]
This is how people get experience with ML though. I don’t think that’s a bad thing.
It sounds like you’re looking for a candidate with current ML experience. But I’ve seen so many people go from zero knowledge to capable devs that this seems like a mistake. You’ll end up overpaying.
Just try to find someone with a burning ambition to learn. That seems like the key to get someone capable in the long run. If they point out something beyond Kaggle that makes you think, pay attention to that feeling — it means they’re in it for more than the money.
I wish there was an easier way to label roles differently based on when you just need to throw X or Y model at some chunk of data and when more specialized modeling is required. Previously it was roughly delineated by "data science" vs "ML" roles but the recent AI thing has really messed with this.
If you're teaching them, you shouldn't be paying them at the AI expert rate.
These days people can get an excellent introductory class to spark and be just as good as I've ever been at it. I wouldn't call them 'charlatans' like the poster above did. It's just that the libraries used to implement spark have been abstracted and people learn it faster.
That's just how it goes in tech. Anyone who wants to learn is a treated like a poser. We over-index on academic credentials which are really not indicative of actual hands-on engineering ability.
PS. There are no AI/ML experts. There are LLM experts, prediction model experts, regression experts, image recognition experts.... If you are hiring a 'AI/ML expert', you have no idea what you are hiring.
But that doesn’t mean that having people with actual research/depthful expertise aren’t essential and hard to find amongst the noise.
The person you responded to is talking about would-be technicians applying for researcher roles. That happens in tech booms and opens amazing doors for lucky smart people, but it’s also a huge PITA for hiring managers to deal with.
One important aspect of all skills is to know the limitations and boundaries of said skill. It’s probably fine if somebody implemented ML on a trivial problem to learn and practice, but if they didn’t realize there could be better solutions in the first place and that ML isn’t a solution to everything, then it’s a big red flag for me.
Also, finding a good problem for a solution is also a handy skill, if one can’t figure out how to apply their skills to a real problem, then that does give slightly negative impressions.
I think the hype on the field and the shitty candidate pool go hand in hand. The shitty candidate pool will groupthink / cargo cult the space without much critical thinking. The groupthink / hype will cause people to jump into the field who don't have any business being in the field.
I've seen two variants of this
1) People that have worked for traditional (as in non-tech) companies, where there's been a huge push for digitalization and "AI". These things come from the very top, and you don't really have much say. I've been there myself.
The upper echelon wants "AI" so that they can tick off boxes to the board of directors. With these folks, its all about managing expectations - but frankly, they don't care if you implement a simple regression model, or spend a fortune on overkill models. The most important part is that you've brought "AI" to the company.
2) The people that want to pad their resumes. There's no need, no push, but no-one is stopping you. You can add "designed and implemented AI products to the business operation blablabla" to your CV.
These days, I've seen and experienced 1) an awful lot. It's all about keeping up with the joneses.
It should be possible for a competent software engineer to get up to speed in AI in less than 6 months and much of that time can be on the job itself.
AI is ill-defined so the premise of your comment makes it difficult to answer. For small well-known tasks (image classification, object detection, sentiment detection) that is train-once on a single dataset and deploy-once what you are saying is true, but for more complex products there is a lot of arcane knowledge that can go in training/deploying/maintaining a model.
On the training side, you need to be able to define the correct metrics, identify bottlenecks in your dataloader, scale to multiple nodes (which is itself a sub-field because distributing a model is not simple) and run evaluation. Throughout the whole thing you have to implement proper dataset versioning (otherwise your evaluation results won't be comparable) and store it in a way that has enough throughput to not bottleneck your training without bankrupting the company (images and videos are not small).
Finally you have a trained model that needs to be deployed, GPU time is expensive so you need to know about compilation techniques/operator fusing, quantization and you need to be able to scale. The requirements to do that are complex because the input data is not always just text.
So yes all the above (and a lot more) require specific expertise.
I know cause I did it.
And I knew the math beforehand. I was a Physics major in college with a CS minor.
I don't think AI is hard to learn. The fundamentals are extremely simple and a competent software engineer can learn all the required concepts in a few months. It's easier if you already have a background in mathematics but not required. If you can write software then you can learn how to write differentiable tensor programs with any of the AI frameworks.
Edit: You asked what it is about these jobs that requires expertise. I answered: it requires expertise to create competitive models. So companies that need competitive models requires expertise.
Edit: Why do you ask? I don't see why it is relevant for the discussion.
Knowing the library is the least hard part about ML work just like knowing the web framework is the least hard part about webdev (both imo). It's much more important to understand the actual problem domain and data and get a smooth data pipeline up and running.
Scaling, optimizing inference, squeezing out better performance and annoying labeling. There's a pretty solid gap from applying some framework to a preexisting and never changing dataset vs. curating said dataset in a changing environment. And if we're talking about RL and not just supervised/unsupervised then building a suitable training environment etc. also become quite interesting.
If someone asked me "what's so hard about webdev" my answer would be similar btw...it's fairly easy to set up a reasonably complicated "hello world" project in any given framework but it gets a lot harder when real world issues like different auth worklflows, security, scaling and handling database migrations etc. enter the picture.
Several years ago on HN there was a blog post which (attempted to) answer this question in detail, and I have been unsuccessfully trying to find it for a long time. The extra facts I can remember about it are:
* It was by a fairly well known academic or industry researcher
* It had reddish graphics showing slices of the problem domain stacking up like slices of bread
* It was on HN, either as a submission or in the comments, between 2016 and 2018.
If anybody knows the URL to this post, I would be stoked!
--
I decided to have a crack myself and see what come back.
Here's a few names / blogs that might be useful:
Chris Olah: He's written extensively about deep learning and AI. His blog, colah.github.io, has a unique graphical style that helps explain complex topics.
Distill.pub: This online journal publishes clear and visually engaging articles on machine learning topics. Some of the articles have been discussed on HN.
Andrej Karpathy: Director of AI at Tesla and previously a researcher at OpenAI and Stanford. He's known for his blog, karpathy.github.io, where he delves into various AI topics.
Ian Goodfellow: Known for inventing Generative Adversarial Networks (GANs) and for his deep learning textbook. He might have some writings that match your description.
Ben Recht: A professor at Berkeley who writes about the challenges and misunderstandings in machine learning on his blog, www.argmin.net.
Sebastian Ruder: He has written many articles about NLP and machine learning at ruder.io.
You can also try searching via https://hn.algolia.com/
But becoming an X-Scientist (Data/Applied/Applied Research) is a whole different skill set. Now, this kind of role only exists in a proper ML company. But, just acquiring the Statistics & Linear Algebra 201 level intuition is about 6 months of fulltime study in its own right. You also need to have deep skills in one of the Tabular/Vision/NLP/Robotics areas and get hired into a role accordingly. Usually 1 year intensive masters level is good enough to get your foot in the door, with the more prestigious roles needing about 2 years of intensive work with some track record of State-of-the-art results on 1 occasion.
Then you have proper researchers, and that might be the most impossible to get in field right now. I know kids who have only done hardcore ML since high school, who are entering the industry after their masters or PhD. I would not want to be an entry level researcher right now. You need to have undergrad math-CS dual major level skills just to get started. They're expected to have delivered state-of-the-art results a few times just to be called for an interview. I'd say you need at least 3 years of fulltime effort if you want to pivot into this field from SWE.
If something is already done, i.e. a model is available for your exact use case (which is never), then for using and deploying that can be done by a good SWE and no ML/AI specialist is not needed at all.
To solve any real problem that is novel, you need to know a lot of things. You need to be on top the progress made by reading papers and be a good enough engineer to implement the ideas that you are going to have iff you are creative/a good problem solver.
And to read those papers you need to have solid college level Calculus and Stats.
If this is so easy, then why don't you do it, and get a job at OpenAI/Tesla/etc?
I've been in industry and now I do research at a top university. I hand pick the best people from all over the world to be part of my group. They need years under expert guidance, with a lot of reading that's largely unproductive, while being surrounded by others doing the same, in order to become competent.
Writing code is easy. You can learn to use any API in a weekend. That's not what is hard.
What's hard is, what do you do when things don't work. Fine, you tried the top 5 models. They're all ok, but your business requirements need much higher reliability. What do you do now?
This isn't research. But you need a huge amount of experience to understand what you can and cannot do, how to define a problem in a way that is tractable, what problems to avoid and how to avoid them, what approaches cannot possibly work, how to tweak and endless list of parameters, how to know if your model could work if you spent another 100k of compute on it or 100k of data collection, etc.
This is like saying you can learn to give people medical advice in 6 months. Sure, when things are going well, you could handle many patients with a Google search. But the problem is what happens when things go badly.
I’m also not surprised by the “The number of candidates applying with claimed ML/AI experience who haven’t done anything more than follow online tutorials is wild”. Just go look at any Ask HN thread about “how do I get into ML/AI”. This is pretty typical advice. Hell it’s pretty typical advice given to people asking how to get into any domain. Now sure we’ll how it works outside of bog standard web development though.
Sure, I get this, but I suspect that the number of people who have actual ML/AI experience is pretty small given that the field is nascent. If you really want to hire people to do this kind of work you're going to need to go with people who have done the online tutorials, read the papers, have an interest, etc. Yes, once in a while you're going to find someone who has actual solid ML experience, but you're also going to have to pay them a lot. That's just how things work in a field like this that's growing rapidly.
You seem to look down on those who have
1) learned from online courses
or
2) used AI on tasks that don't require it
Isn't this a bit contradictory? Or you expect candidates to have found a completely novel usecase for AI on their own?
I understand that most ML roles prefer a master's degree or PhD, but from my experience most of the master's degrees in ML being offered right now were spawned from all the "AI hype". That is to say, they may not include a lot of core ML courses and probably are not a significantly better signal of a candidate's qualifications than some of the good online courses out there.
So what does that leave, only those with a PhD? I think it's unreasonable that someone should need that many years of formal education to get an entry level position. Maybe I'm missing something, but I'm really wondering, what do you expect from candidates? I think a few years of professional software engineering experience with some demonstrated interest in AI via online courses and personal projects should be enough.
Most companies doing regular, non-ML development hire a mix of junior and experienced engineers, with the latter providing code reviews, mentorship and architectural advice alongside normal programming duties.
It's understandable that someone kicking off a new ML project would hope to get the experienced hires on board first.
But there are a lot more junior people on the market than senior people right now - as is the nature of a fast growing market.
I agree, it's problematic that there are so many more juniors than seniors in the industry right now. I feel like many juniors are being left without mentorship, and then it becomes much harder for them to grow and eventually become qualified for senior roles. So that could help explain why many candidates seem so weak, alongside with all the recent hype.
I guess eventually the market will cool off and the hype will die down since this stuff seems to be cyclical, and the junior engineers who are determined enough to stick it out and seek out mentorship will be able to grow and become seniors.
But it definitely seems like the number of seniors is a bottleneck for talent across the industry.
TBF theres whole companies doing this. It's a good way to learn too, as you have existing solutions to compare yourself too.
Got a senior ML position in a well known Fortune 500 company. Senior enough that he sets his goals - no one gives him work to do. He just goes around asking for data and does analyses. When he left our team he told me "Now that I have this opportunity, I can actually really learn ML instead of faking it."
If you think that's bad, you should hear the stories he tells at that company. Since senior leadership knows nothing about ML practices, practices are sloppy to get impressive numbers. Things like reporting quality based on performance on training data. And when going from a 3% to a 6% prediction success rate, they boast about "doubling the performance".
He eventually left for another company because it was harder to compete against bigger charlatans than he was.
If he really did take those and did all the assignments himself and understood all the concepts, that still puts him at least in the 95th percentile among ML job seekers.
But it shows intent for junior roles.
People have no idea how many people just put "AI enthusiast" on LinkedIn profile and start to seek ML roles.
I have had people only with Excel skills apply to ML roles.
And, true thing is every company’s ML tooling/stack/procedure is wildly different. One has to learn on the job.
(Shrug) I don't. Hustle gets rewarded, as usual. Sounds like he contributed at least as much value as he captured.
What in my comment gave you that idea?
Oh, and he didn't have to write any code during the interview.
You don't see anything wrong with that?
I think it's fine to hire a person who just took Coursera ML courses and passes the interview, but you would normally position the person to be a junior with senior folks overseeing his work.
Yeah this is a major phenomenon. Everybody's putting "ai" stickers on everything. So the job market screams "we need ai experts!" in numbers far exceeding the supply of ai experts, because it was a tiny niche until a couple years ago. Industry asks for garbage, industry gets garbage.
The applied research ML role has evolved from being a computational math role to a Pytorch role to a 'informed throw things at the wall' role.
I went from reading textbooks (Murphy, Ian Goodfellow, Bishop) to watching curated NIPS talks to reading Axriv papers to literally trawling random discord channels and subreddits to get a few month leg up on anyone in research. Recently, a paper formally cited /r/localllama for their core idea.
> follow online tutorials
The Open Source movement moves so quickly, that running someone's collab notebook is the way to be at the cutting edge of research. The entire agents, task planning and meta-prompting field was invented in random forums.
________________
This is mostly relevant to the NLP/Vision world......but take a break for 1-2 years, and your entire skill set is obsolete.
You probably do not need AI experts if you just need good machine learning engineer to build models to solve problems.
Then again, that seems to be common with the job market.
And out of all the nightmare “we have so many qualified candidates we can’t even do price discovery” conversations in 2023, the ML ones have been the worst.
If you’re running a serious shop that isn’t screwing around and you’re having trouble finding tenured pros who aren’t screwing around, email me! :)
This is not specific to ML/AI roles. The same problem applies to anyone who wants to explore any of these domains - SRE, DataEng, Backend, Frontend.
Personally, I am a backend engineer who wants to get into ML Infra roles. My current plan is to do these online courses, and hopefully transfer internally to a team working in this area. Only after having real industry experience in this area, look for more opportunities elsewhere.
I am genuinely curious if anyone has better ideas for people in my situation.
I did some hiring for a very real machine learning (AI if you want to call it that) initiative that started even before the LLM explosion. The number of candidates applying with claimed ML/AI experience who haven’t done anything more than follow online tutorials is wild. This was at a company that had a good reputation among tech people and paid above average, so we got a lot of candidates hoping to talk their way into ML jobs after completing some basic courses online.
The weirdest trend was all of the people who had done large AI projects on things that didn’t need AI at all. We had people bragging about spending a year or more trying to get an AI model to do simple tasks that were easily solved deterministically with simple math, for example. There was a lot of AI-ification for the sake of using AI.
It feels similar to when everyone with a Raspberry Pi started claiming embedded expertise or when people who worked with analytics started branding themselves as Big Data experts.