The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I described usually had a STEM type PhD.
Basically, everyone has experience when they graduate from a master or a PhD there.
Not all experiences are born equal. e.g. Some work is at top tech companies, some work is at universities or low quality research centers with little work & expectations.
I am curious what countries it works the other way around.
But that's by no means all of the DS field. There are lots of DS jobs where you're collecting and interpreting and communicating about complex data sets. An engineering mindset is occasionally helpful, but a bias towards building versus towards analyzing and writing can just as often be counter-productive. Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution.
The bootcamps have contributed to the problem by focusing so much on building things. The idea that you can go from an econ undergrad to being a self-sufficient member of a production ML team in 6-12 weeks is nuts. What's less nuts (and what I wish programs like Insight focused on) is taking people from having data skills in one domain and with one set of tools (e.g. logitudinal medical record data, stored in CSVs and handled in Stata) to another set of tools (billions of rows of event-based product data stored in a data warehouse, processed in R or Python). But instead the bootcamps behave like the missing skillset is the ability to make a predictive random forest model on some arbitrary data set and build an AWS web app around it. THAT job market definitely doesn't exist and is completely over-saturated.
But people who are smart communicators about data, can manipulate and make sense of massive data sets, can ask incisive questions about their data, and can use data to convince people of a complex argument are always going to have job opportunities, even if they're not production grade engineers. If that sounds like you, I'm hiring - hit me up on Twitter: @drewwww.
Agreed wholeheartedly. Reminds me of another quote from http://www.john-foreman.com/blog/surviving-data-science-at-t... :
""" You know what can keep up with a rapidly changing business?
Solid summary analysis of data. Especially when conducted by an analyst who's paying attention, can identify what's happening in the business, and can communicate their analysis in that chaotic context.
Boring, I know. But if you're a nomad living out of a yurt, you dig a hole, not a sewer system. """
I am happily serving a niche market with my own company and I suspect part of the difficulty in finding the skillset is that we can just start our own thing when we find a domain that we like.
I'm not coming from a CS background and don't purport to know absolutely all of the details of all the mathematics and theory behind many of the machine learning algorithms that I use. I try my best daily to expand my knowledge, understand the algorithms, and apply them appropriately. I would hope that any company who is looking for someone who has a PhD-in-CS-or-ML could weed someone with lesser knowledge out during the interview process.
With that being said, a couple lines of code using SciKit Learn and all the default parameters is enough to impress many non-tech companies that are looking for a way to use 'predictive' in their marketing materials. And they pay very well for it. I get the feeling that provokes the ire of people who think those types of basic implementations belong to the traditional label of 'data analyst'.
For what it's worth, I work with data sets that aren't quite large enough to justify anything more than Python, Pandas, SKLearn, Luigi pipeline, and PySpark. The vast majority of my time is spent cleaning the data and generating features, must less on the hyperparameter tuning, model training side itself.
Anyways, I think I'm rambling a bit.
I just want to say that I LOVE this job, whatever the label is, or whatever the hype surrounding the label is, and I hope it's around for a while before it's automated...
Except for when it's not - those are the cases you're hired for.
But if you have an important data driven decision that is going to cost you a lot of resources (money, people, time, etc), you want the analyst to be able to speak to what the data and/or model is telling you and why. When you have this type of problem and it's "solved" by a few lines of SciKit Learn, you should be prepared to have a really bad day at some point in the future. This is the type of Data Scientist that draws the ire of people who do analysis with more depth.
Just find who's looking for economists, specifically. Consulting firms and government agencies hire economists, but they don't call them data scientists. Investment funds may be interested, though hedge funds prefer math and physics graduates because they are easier to train.
Forget about the big companies that everyone knows, forget about the high pay that appear in the newspapers. It's not real, it's not for you.
Get an entry or junior level position and get experience. There are many unknown places that will take people only because they're cheap. That's where everyone started.
- excellence in being able to clean data at the column level for millions of data points - knowing how to work with such large scale data in a time efficient manner. One of our newer hires worked at the Postgres/SSD level to optimize and got it to where he could produce a full set within 5 minutes. Before that it was once every few months.
Being able to do these things is a prerequisite to building even a prototype of a model, and it requires substantial real-world programming experience to deal with those complexities.
- Research departments that make prototypes that are consistently abandoned and never get any real world usage.
- Actual companies that make prototypes for real business cases that are then shipped to production, maintained and improved as they bring in $$$.
The first experience is of limited value to the world of real business.
I'm continually exposed to new kinds of software engineering roles I never heard of at tech companies. (fwiw software engineer is sometimes still considered an inflated title.)
One data engineering role asked me to implement k-means from scratch and one data science role asked me to do some algorithms whiteboarding. But beyond this, people just asked SQL. From the rest of this thread it doesn't look like people would consider that to be 'SWE ability'.
Should data scientists be expected to do more than that? Or are you expecting a certain level of code quality from them? Within a 48 hour time window?
3 easy steps to get a job in DS if you want them though: Grad Degree in Math/Stats/CompSci; work on a bunch of hard to predict problems and then publish and present them to your local meetup community to gain experience; learn engineering tools and devops and be about 90% as good a software engineer as your team's actual engineers (git, hg, IDEs, java, pig)... your brilliant models are way less important than being able to help the already overwhelmed engineering team make them work.
Eh? It's not as easy as it sounds. Search for "mlelr" for a rather detailed illustration of how to code LR in C with only standard libraries. Now that was some fun to put together!
(Edit: I'm not saying MOOC-depth derisively. Serious learners and autodidacts can go pretty deep with just MOOCs for guidance. I just mean prima facie, based on content of some of the lighter and more popular MOOCs like Andrew Ng's. Abu-Mostafa's on EdX is meatier.)
Anyone can be quickly taught how to run a logistic regression by calling one line of code from a high level library. But I'd argue it takes years of study to really know what you're doing.
Could you elaborate on which elements of vision science are relevant? Do you have any examples of what these people end up working on?
Would an English MA or PhD, for example couple well with an ML-MOOC for natural language processing jobs? Possibly, but would involve a lot of effort to build the bridge. However, if someone has a mechanical engineering degree or a PhD in something similar, a MOOC-ML can work well to position them for a bunch of jobs in large mechanical engineering companies looking to jump on the ML bandwagon or ones traditionally strong in ML-type mechanical engineering like UTC.
As far as this thread, don't worry too much about it.
The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it.
The second group has gotten extremely crowded, especially from STEM graduates - usually with a masters or phd - who have completed MOOCs or bootcamps. Supply keeps growing while demand is flat or shrinking (especially as executives get burned by "data scientists" who don't know how to help them build things of value). There's a huge crunch here; a lot of people I know in this group have been searching for jobs for months, eventually settling for a low quality job or giving up entirely :(