News for hackers. Stuff that matters.
Re:Ask HN: What's the state of the job market in data scienc... by rm999 on Wednesday December 21, 2016 @05:50PM (#13233352)
Speaking for NYC, but I imagine silicon valley is similar.

The supply-demand dynamics have changed a lot in the last couple years. I'd roughly break it out into two groups: people with work experience + strong software development skills, and those without. The first group is in higher demand than ever, and tend to add a lot of value to companies that really need it.

The second group has gotten extremely crowded, especially from STEM graduates - usually with a masters or phd - who have completed MOOCs or bootcamps. Supply keeps growing while demand is flat or shrinking (especially as executives get burned by "data scientists" who don't know how to help them build things of value). There's a huge crunch here; a lot of people I know in this group have been searching for jobs for months, eventually settling for a low quality job or giving up entirely :(

Re:Ask HN: What's the state of the job market in data scienc... by pcsanwald on Wednesday December 21, 2016 @06:03PM (#13233436)
I've only been hiring DS folks since 2012, but my experience matches what you've said exactly. The biggest differentiator I've seen is to be able to participate in actually building production quality systems vs being proficient enough in R or python to hack together a prototype on a very small dataset.

The former kind of data scientists were very successful at our company, the latter, not so much. Both categories I described usually had a STEM type PhD.

Re:Ask HN: What's the state of the job market in data scienc... by user5994461 on Wednesday December 21, 2016 @06:06PM (#13233454)
That sounds weird to me. Does American PhD don't have to work for a few years at real companies as part of the PhD curriculum?
Re:Ask HN: What's the state of the job market in data scienc... by ylem on Wednesday December 21, 2016 @06:12PM (#13233480)
I think the scenario they are describing is for say math/physics PhDs who are transitioning to data science. During their degrees, they concentrate on research, so they don't (on average) have real software engineering experience.
Re:Ask HN: What's the state of the job market in data scienc... by sigstoat on Wednesday December 21, 2016 @07:46PM (#13233921)
no. it's even a stereotype that they've never worked at a company. where is it required that PhDs work at companies to receive a degree?
Re:Ask HN: What's the state of the job market in data scienc... by user5994461 on Thursday December 22, 2016 @06:01PM (#13241187)
France. There are various internships dues during the PhD and the final paper is done while working 1-3 years at one place.

Basically, everyone has experience when they graduate from a master or a PhD there.

Not all experiences are born equal. e.g. Some work is at top tech companies, some work is at universities or low quality research centers with little work & expectations.

Re:Ask HN: What's the state of the job market in data scienc... by gumby on Wednesday December 21, 2016 @10:41PM (#13234598)
I am used to the PhD being an academic, not practice (trade) degree. This is true even in fields where the PhD is pretty much required for work in industry (medicinal chemistry, biology etc). And even in computer science where a PhD can even be a hindrance to getting an industry job, the PhD is pretty much university only.

I am curious what countries it works the other way around.

Re:Ask HN: What's the state of the job market in data scienc... by drew on Wednesday December 21, 2016 @09:03PM (#13234214)
The problem is DS is really 2-3 different disciplines under one nebulous title. What you're describing is folks who are prototyping and productionizing models. That's definitely in short supply, but random STEM PhDs are in no way competitive for those roles unless they're coming from CS programs + have work experience in production engineering.

But that's by no means all of the DS field. There are lots of DS jobs where you're collecting and interpreting and communicating about complex data sets. An engineering mindset is occasionally helpful, but a bias towards building versus towards analyzing and writing can just as often be counter-productive. Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution.

The bootcamps have contributed to the problem by focusing so much on building things. The idea that you can go from an econ undergrad to being a self-sufficient member of a production ML team in 6-12 weeks is nuts. What's less nuts (and what I wish programs like Insight focused on) is taking people from having data skills in one domain and with one set of tools (e.g. logitudinal medical record data, stored in CSVs and handled in Stata) to another set of tools (billions of rows of event-based product data stored in a data warehouse, processed in R or Python). But instead the bootcamps behave like the missing skillset is the ability to make a predictive random forest model on some arbitrary data set and build an AWS web app around it. THAT job market definitely doesn't exist and is completely over-saturated.

But people who are smart communicators about data, can manipulate and make sense of massive data sets, can ask incisive questions about their data, and can use data to convince people of a complex argument are always going to have job opportunities, even if they're not production grade engineers. If that sounds like you, I'm hiring - hit me up on Twitter: @drewwww.

Re:Ask HN: What's the state of the job market in data scienc... by ikding on Thursday December 22, 2016 @09:15AM (#13237338)
"Not all problems are solved by systems; lots of problems are solved by better understanding the problem and then letting other specialists build the right solution."

Agreed wholeheartedly. Reminds me of another quote from http://www.john-foreman.com/blog/surviving-data-science-at-t... :

""" You know what can keep up with a rapidly changing business?

Solid summary analysis of data. Especially when conducted by an analyst who's paying attention, can identify what's happening in the business, and can communicate their analysis in that chaotic context.

Boring, I know. But if you're a nomad living out of a yurt, you dig a hole, not a sewer system. """

Re:Ask HN: What's the state of the job market in data scienc... by ma2rten on Wednesday December 21, 2016 @10:41PM (#13234597)
I was part of the Data Science team in my previous company. We mainly build models for production, but we also were responsible for generating both daily and ad hoc reports. We tried to hire someone to take over the reporting part, but we found out even that requires engineering skills. This role ended up to even more difficult to hire for because it's hard to find someone who has the engineering skills but wants to work only on reporting. Maybe if we had a dedicated data engineering team the story would be different.
Re:Ask HN: What's the state of the job market in data scienc... by jaxn on Wednesday December 21, 2016 @11:21PM (#13234781)
That is me. A decent developer who gets data and enjoys reporting, particularly the variety of hard problems that crop up.

I am happily serving a niche market with my own company and I suspect part of the difficulty in finding the skillset is that we can just start our own thing when we find a domain that we like.

Re:Ask HN: What's the state of the job market in data scienc... by drew on Thursday December 22, 2016 @04:07AM (#13235962)
Yeah, in my org we have a dedicated data engineering team that owns the pipeline and production data systems plus an analytics team that owns reporting and making data available to non-technical data consumers. That leaves complex research work for data scientists who are (in theory, anyway) building on top of stable data infrastructure and good data sources and free from ad hoc reporting work.
Re:Ask HN: What's the state of the job market in data scienc... by curiousgal on Thursday December 22, 2016 @04:27AM (#13236047)
In France, and other Francophone countries, Statistics and Information Analysis is an engineering discipline.
Re:Ask HN: What's the state of the job market in data scienc... by wakkaflokka on Thursday December 22, 2016 @09:12AM (#13237325)
I'm coming from a PhD in STEM where I did a lot of application of basic ML to my (neuroscience) research, and it took me a good while to get a data science position. But once I got the job, I have been inundated with interview requests, both from recruiters and from specific companies, as in multiple a week (granted most are blast-em-all style recruiters, that I'm sure anybody with any tech skills get). Maybe it's because I'm on the East Coast? I do feel like everybody in neuro is jumping on the bandwagon, and that generates a bit of "you don't belong here" feeling from the CS- or math- educated crowd. But over time I think this will all level-out.

I'm not coming from a CS background and don't purport to know absolutely all of the details of all the mathematics and theory behind many of the machine learning algorithms that I use. I try my best daily to expand my knowledge, understand the algorithms, and apply them appropriately. I would hope that any company who is looking for someone who has a PhD-in-CS-or-ML could weed someone with lesser knowledge out during the interview process.

With that being said, a couple lines of code using SciKit Learn and all the default parameters is enough to impress many non-tech companies that are looking for a way to use 'predictive' in their marketing materials. And they pay very well for it. I get the feeling that provokes the ire of people who think those types of basic implementations belong to the traditional label of 'data analyst'.

For what it's worth, I work with data sets that aren't quite large enough to justify anything more than Python, Pandas, SKLearn, Luigi pipeline, and PySpark. The vast majority of my time is spent cleaning the data and generating features, must less on the hyperparameter tuning, model training side itself.

Anyways, I think I'm rambling a bit.

I just want to say that I LOVE this job, whatever the label is, or whatever the hype surrounding the label is, and I hope it's around for a while before it's automated...

Re:Ask HN: What's the state of the job market in data scienc... by rcpt on Thursday December 22, 2016 @10:41AM (#13237952)
> a couple lines of code using SciKit Learn and all the default parameters is enough

Except for when it's not - those are the cases you're hired for.

Re:Ask HN: What's the state of the job market in data scienc... by apohn on Thursday December 22, 2016 @12:23PM (#13238723)
There are also no shortage of "Data Science" projects that are basically just showing off (e.g. somebody in a large org trying to get recognition for being advanced) or using an algorithm to justify a decision that somebody had already decided to make. In this case, it's okay to solve the problems with a few lines of code and a nice presentation.

But if you have an important data driven decision that is going to cost you a lot of resources (money, people, time, etc), you want the analyst to be able to speak to what the data and/or model is telling you and why. When you have this type of problem and it's "solved" by a few lines of SciKit Learn, you should be prepared to have a really bad day at some point in the future. This is the type of Data Scientist that draws the ire of people who do analysis with more depth.

Re:Ask HN: What's the state of the job market in data scienc... by DrNuke on Thursday December 22, 2016 @01:17PM (#13239148)
Let's be blunt: firms needing real data science know where to go and find real data scientists. Anything less real, on both sides, is matter of opinion and here it comes free market.
Re:Ask HN: What's the state of the job market in data scienc... by cardine on Wednesday December 21, 2016 @07:13PM (#13233766)
Even outside Silicon Valley this is very similar. When I post a programming job I get more applicants from data scientists than from actual programmers. When I post a marketing job I get more applicants from data scientists than from actual marketers. It is clear to me that supply has far outpaced demand.
Re:Ask HN: What's the state of the job market in data scienc... by dnautics on Wednesday December 21, 2016 @07:33PM (#13233862)
I'm getting burned because I have a STEM PhD and am a competent programmer (have been programming on the side for decades) and am looking for a not-necessarly-data-science job, but it seems like all the places I'm applying to want to bin me in the "data-science" category.
Re:Ask HN: What's the state of the job market in data scienc... by hellogoodbyeeee on Wednesday December 21, 2016 @06:17PM (#13233509)
Do you have advice for someone in the second camp? I have a master's in econ and I'm comfortable in R but I don't have good development skills. I'm wondering if I need to give up on data science jobs for a bit and try and find an entry level software development job
Re:Ask HN: What's the state of the job market in data scienc... by anton_tarasenko on Wednesday December 21, 2016 @06:36PM (#13233595)
Target research-intensive jobs. Job descriptions are clear about what sort of machine learning professionals the company is looking for. In most cases, they want data engineers and developers who can make big, scalable production. Not researchers.

Just find who's looking for economists, specifically. Consulting firms and government agencies hire economists, but they don't call them data scientists. Investment funds may be interested, though hedge funds prefer math and physics graduates because they are easier to train.

Re:Ask HN: What's the state of the job market in data scienc... by user5994461 on Wednesday December 21, 2016 @06:41PM (#13233616)
Take whatever job you can... as far as it's related to the kind of things you want to do.

Forget about the big companies that everyone knows, forget about the high pay that appear in the newspapers. It's not real, it's not for you.

Get an entry or junior level position and get experience. There are many unknown places that will take people only because they're cheap. That's where everyone started.

Re:Ask HN: What's the state of the job market in data scienc... by hellogoodbyeeee on Wednesday December 21, 2016 @06:48PM (#13233654)
Thanks for your comment. It has eased some of the anxiety caused by reading through this thread. My current job fits your description. I currently have an analyst role and I spend maybe 25% of my time building predictive models in R. I just haven't had any success finding a role that ups that percentage (and the pay). I know I need to be more patient, but it is difficult.
Re:Ask HN: What's the state of the job market in data scienc... by nether on Wednesday December 21, 2016 @08:27PM (#13234076)
I'm looking to go through a DS boot camp this summer, have a master's in aerospace engineering. I'd be very content with mid-five figures (40% lower than what I made as an engineer) at a no-name company if it's in the city that I want (not SF). Are such lower-end data analyst positions in decent supply?
Re:Ask HN: What's the state of the job market in data scienc... by TXV on Wednesday December 21, 2016 @06:01PM (#13233428)
I think what you say is easily applicable to software engineering in general. Data science maybe is a field that is even more negatively impacted by bad hires because the threshold after which you start adding value to the company is higher.
Re:Ask HN: What's the state of the job market in data scienc... by hiddencost on Thursday December 22, 2016 @12:35AM (#13235114)
This is correct. I've worked in industry for about 3 years, much in a top industry lab that built a product you probably use. I've got a couple papers, including a really good one. I've worked as a software engineer. I hopped jobs twice in a relatively short frame, with a 20% and 40% raise each time. The market is on fire if you have software engineering competence and real industrial experience.
Re:Ask HN: What's the state of the job market in data scienc... by rafinha on Wednesday December 21, 2016 @06:26PM (#13233552)
Isn't it 2 profiles of engineers? Those who make production code and those who work in prototypes? One thing is to understand why your model isn't converging, another is how to scale it up...
Re:Ask HN: What's the state of the job market in data scienc... by audleman on Wednesday December 21, 2016 @06:37PM (#13233597)
A machine learning model is core to my company, and core to that is pulling in, cleaning, and working with large dataset from a variety of platforms. This means:

- excellence in being able to clean data at the column level for millions of data points - knowing how to work with such large scale data in a time efficient manner. One of our newer hires worked at the Postgres/SSD level to optimize and got it to where he could produce a full set within 5 minutes. Before that it was once every few months.

Being able to do these things is a prerequisite to building even a prototype of a model, and it requires substantial real-world programming experience to deal with those complexities.

Re:Ask HN: What's the state of the job market in data scienc... by user5994461 on Wednesday December 21, 2016 @06:48PM (#13233651)
2 profiles:

- Research departments that make prototypes that are consistently abandoned and never get any real world usage.

- Actual companies that make prototypes for real business cases that are then shipped to production, maintained and improved as they bring in $$$.

The first experience is of limited value to the world of real business.

Re:Ask HN: What's the state of the job market in data scienc... by elastic_church on Wednesday December 21, 2016 @06:50PM (#13233659)
the data science people I've worked with in multiple VC backed companies were not engineers at all. They were either "data scientists" or had engineer tacked on to the name.

I'm continually exposed to new kinds of software engineering roles I never heard of at tech companies. (fwiw software engineer is sometimes still considered an inflated title.)

Re:Ask HN: What's the state of the job market in data scienc... by huac on Wednesday December 21, 2016 @07:37PM (#13233884)
How do you test for programming ability in an interview? Especially for people without a lot of work experience (e.g. graduating from college)

One data engineering role asked me to implement k-means from scratch and one data science role asked me to do some algorithms whiteboarding. But beyond this, people just asked SQL. From the rest of this thread it doesn't look like people would consider that to be 'SWE ability'.

Re:Ask HN: What's the state of the job market in data scienc... by nicoburns on Wednesday December 21, 2016 @07:49PM (#13233932)
Set someone the interviewee a programming task. This is pretty standard for developer jobs, and is absolutely fantastic for people without work experience (as it gives them another way to probe their skill).
Re:Ask HN: What's the state of the job market in data scienc... by huac on Wednesday December 21, 2016 @09:06PM (#13234216)
I mean, I did a bunch of take-home tests. But those were focused on doing an analysis on some dataset, focused on answering a question.

Should data scientists be expected to do more than that? Or are you expecting a certain level of code quality from them? Within a 48 hour time window?

Re:Ask HN: What's the state of the job market in data scienc... by bjlkwpZxAygCIg on Tuesday December 27, 2016 @06:09PM (#13267071)
There's always room for smart, hard working and creative thinkers in any field. Except maybe marine biology. But as in any lucrative field, e.g. law, you have a wide variety of capabilities and work quality. After 10 years in quantititative work (I got an MS in Applied Math before they were calling Statistics "Data Science") I can say that it's just a tough field to work in. Experience goes further than education, but education (yes, a real accredited graduate level education) is necessary. The most challenging aspect of DS isn't the technical aspects, it's being able to have a thick enough skin to not let the skeptical and reluctant engineering types upon who you depend to implement your brilliant models get under your skin and enough patience to explain and convince the skeptical and not too savvy business types who cut your checks, that their intuition is wrong and your math is right. And then of course, there's the inevitable boredom that comes with solving yet another mundane business problem with the simplest and least sexy tools. I'm not complaining, just saying that almost all STEM graduates turned DS I've ever worked with have a small hollow spot in their soul that burns with passion for the astrophysics or theoretical math problems they traded in for the perk filled corporate life.

3 easy steps to get a job in DS if you want them though: Grad Degree in Math/Stats/CompSci; work on a bunch of hard to predict problems and then publish and present them to your local meetup community to gain experience; learn engineering tools and devops and be about 90% as good a software engineer as your team's actual engineers (git, hg, IDEs, java, pig)... your brilliant models are way less important than being able to help the already overwhelmed engineering team make them work.

Re:Ask HN: What's the state of the job market in data scienc... by shas3 on Wednesday December 21, 2016 @06:20PM (#13233517)
Frankly, MOOC-type machine learning- being able to do plain vanilla logistic regression or black-box deep learning techniques is not enough to get a job. This knowledge and experience has to be paired with one or more strengths: excellence in programming, grad-level math/stat/numerical skills/theoretical machine learning, domain specific expertise or experience (e.g. vision, audio, natural language, networks, geophysics, biomed, finance, etc.), proven ability to learn and adapt very fast.
Re:Ask HN: What's the state of the job market in data scienc... by czep on Wednesday December 21, 2016 @07:42PM (#13233903)
> plain vanilla logistic regression

Eh? It's not as easy as it sounds. Search for "mlelr" for a rather detailed illustration of how to code LR in C with only standard libraries. Now that was some fun to put together!

Re:Ask HN: What's the state of the job market in data scienc... by shas3 on Wednesday December 21, 2016 @07:45PM (#13233915)
But that does mean that you have pretty good C programming skills! I think your comment reinforces my claim about pairing MOOC-depth ML with other skills.

(Edit: I'm not saying MOOC-depth derisively. Serious learners and autodidacts can go pretty deep with just MOOCs for guidance. I just mean prima facie, based on content of some of the lighter and more popular MOOCs like Andrew Ng's. Abu-Mostafa's on EdX is meatier.)

Re:Ask HN: What's the state of the job market in data scienc... by czep on Thursday December 22, 2016 @09:48AM (#13237531)
I definitely agree: the overuse of techniques like logistic regression in introductory courses without sufficient mathematical background and context can lead to a "plug and chug" culture where the technique is applied without proper understanding of its assumptions and limitations.

Anyone can be quickly taught how to run a logistic regression by calling one line of code from a high level library. But I'd argue it takes years of study to really know what you're doing.

Re:Ask HN: What's the state of the job market in data scienc... by omginternets on Wednesday December 21, 2016 @06:43PM (#13233627)
> vision

Could you elaborate on which elements of vision science are relevant? Do you have any examples of what these people end up working on?

Re:Ask HN: What's the state of the job market in data scienc... by shas3 on Wednesday December 21, 2016 @06:53PM (#13233671)
I loosely used the term to refer to computer vision, which overlaps a bit with vision science (especially color, perception, etc.). But yeah, computer vision.
Re:Ask HN: What's the state of the job market in data scienc... by ywfs3370 on Saturday December 24, 2016 @05:17AM (#13249254)
So... for instance an MD with MOOC-type machine learning could make it in health related enterprises? Is that what you mean by domain specific expertise?
Re:Ask HN: What's the state of the job market in data scienc... by shas3 on Tuesday December 27, 2016 @04:17PM (#13266258)
That's stretching it. I think I should have qualified domain specific expertise as quantitative-domain specific expertise. Afaik, MD is more about human biology and meant to help you become a practicing doctor. PhD-biomedicine, MPH/MS-bioinformatics, or MD-PhD programs tend to have more substantial quantitative components than just an MD. Caveat: these are gross generalizations. I know some MDs who are competent enough in quantitative analysis to give any engineer a run for their money!

Would an English MA or PhD, for example couple well with an ML-MOOC for natural language processing jobs? Possibly, but would involve a lot of effort to build the bridge. However, if someone has a mechanical engineering degree or a PhD in something similar, a MOOC-ML can work well to position them for a bunch of jobs in large mechanical engineering companies looking to jump on the ML bandwagon or ones traditionally strong in ML-type mechanical engineering like UTC.

Re:Ask HN: What's the state of the job market in data scienc... by datascientist on Thursday December 22, 2016 @01:53PM (#13239422)
A few months ago Mikio Braun wrote a great post on this topic (the need for data scientists with strong software development skills) https://www.oreilly.com/ideas/what-is-hardcore-data-science-...
Re:Ask HN: What's the state of the job market in data scienc... by LeanderK on Wednesday December 21, 2016 @07:38PM (#13233887)
I am currently studying Bachelor CS an am interested in ML/DS. How can i make sure i fall in the first group? I try to do some side-projects to expand my skill set and practice software dev. skills, but this whole thread seems pretty discouraging.
Re:Ask HN: What's the state of the job market in data scienc... by jps359 on Wednesday December 21, 2016 @10:02PM (#13234426)
If your perspective of ML was based solely on the comments in this thread you'd think there's 0 opportunity in the field -- obviously not true. The people who comment on these kinds of posts tend to always be incredibly cynical, and I am not sure why. They offer a limited perspective on the industry and you should not let it discourage you from your desire to learn and become an ML engineer. If I were an undergraduate CS student interested in ML/DS I would start by taking an intro course for ML and creating a side project you're proud of.
Re:Ask HN: What's the state of the job market in data scienc... by RA_Fisher on Wednesday December 21, 2016 @07:50PM (#13233939)
Given you're strong in CS, study as much statistical theory as possible. That'd be my advice. You need to be a great programmer, but statistics is what's really in short supply.
Re:Ask HN: What's the state of the job market in data scienc... by aub3bhat on Wednesday December 21, 2016 @08:26PM (#13234070)
Depending on where you are studying you can take ML courses. However the easiest way is to get a Master's degree in CS from Stanford, GA Tech, Cornell, CMU, etc. Few Universities offer a Data Science Masters degree e.g. I think NYU, Columbia & Harvard. You can also try joining a comapny and then starting the degree program a year later.

As far as this thread, don't worry too much about it.