Re:Becoming a data scientist might be easier than you think
by rm999 on Monday October 15, 2012 @10:38AM
(#4655219)
Meh. I've said it before, and I'll say it again: those contests aren't necessarily a good indicator of who will be a good data scientist, in the same way programming contests tell you little about who will be a good software engineer.
Being a good data scientist requires a lot more than machine learning, including a solid understanding of the business side (deep domain knowledge), the ability to write production-grade software and tools, scripting/hacking/data munging, math/statistics, and common sense. Running a sanitized dataset through machine learning algorithms is maybe 5-10% of it.
I'm not trying to discourage people, I'm thrilled so many people are taking an interest in data sciences and I want to push interested people in a direction where they can excel at it. But this article is dangerous - becoming a data scientist requires a lot of hard work. I've seen a large sample of people (through interviews) who think a single online class is enough to get into the field. It's a great start, but if you want to be valuable you need a wider set of skills.
Re:Becoming a data scientist might be easier than you think
by Quizz on Monday October 15, 2012 @12:01PM
(#4655642)
I think the context and purpose of the article is more like this: Coursera + Kaggle performance is a legitimate a way to identify promising data scientists. As a hiring manager, I would be as interested in talking to these competitors as talking to a kid who just received his BS in computer science with a couple of machine learning/statistic classes under his belt.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @10:55AM
(#4655300)
All the evidence from kaggle indicates that deep domain knowledge is not required. Jeremy Howard has some youtube videos discussing this. Pretty much all the skills you outlined (except for production grade code - which is a software engineer problem, not a data scientist problem) are covered by the contest.
Re:Becoming a data scientist might be easier than you think
by svasan on Monday October 15, 2012 @11:16AM
(#4655399)
Winning a kaggle contest and how a particular statistical model performed under normal business conditions are totally different. (metrics of interest for evaluating the model - how much money was saved, did the model bring about requisite behavior change, etc.) And performance under real life business conditions is what matters, not who won the contest. And to get good models for a specific business need, you do need domain knowledge.
Does kaggle publish how the models performed under normal business conditions?
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @11:23AM
(#4655432)
I'm not following your line of reasoning. Everything is data at the end of the day. All you're doing is creating a predictive model. If the business conditions change, you wold simply send it to the data scientists to reformulate a new model. That's no different to sending it to kaggle again.
Re:Becoming a data scientist might be easier than you think
by svasan on Monday October 15, 2012 @11:35AM
(#4655488)
>> Everything is data at the end of the day.
But you have to interpret the data within the context of the business need/requirement.
Building a credit risk model is vastly different from building a (personal) insolvency/bankruptcy model though both may entail the same set of steps in developing the model. The variables that make it to the model depend on the business need.
In kaggle, one of the datasets that I messed around with had variable labels as Var_1, Var_2, ..., Var_X. So while fitting a model, I would not know why a particular variable made it into the model. You can see that this kind of variable labeling does not give me any insight into how that variable was generated. I need to know whether the variable was raw/aggregated/transformed etc. And that takes you back to understanding the data in the context of the business/domain.
Re:Becoming a data scientist might be easier than you think
by svasan on Monday October 15, 2012 @12:49PM
(#4655897)
A data dump would not really help because the data could have been influenced by
1) a key company policy
2) specific business activity
3) input coming from another model
It is better to
a) define the problem,
b) collect the data,
c) build the variable library,
d) and then fit the model
rather than jump to step (d) directly because the modeler/scientist has greater understanding of the entire set of data going into the model development. It is very likely that the modeler would uncover any/all of the three influencing factors I mentioned above, during the data collection stage.
While kaggle is an interesting concept, from a different perspective it looks like an "effort harvesting" operation. For a pittance, the companies/institutions that are sponsoring the contests are getting a steal. (I am not sure if the million dollar prize is still up for grabs.) However, for folks who do want to break into data sciences/statistics field, kaggle certainly is a good platform to get acquainted with data science/statistics related skills.
Re:Becoming a data scientist might be easier than you think
by micro_cam on Tuesday October 16, 2012 @10:56AM
(#4660078)
Proper featurization is a hugh part of working with real data. Datasets in contests tend to have already been analyzed enough to remove all features that are correlated, caused by or predictive of but irrelevant to the target feature.
For example, early analysis of a "fresh" data set involves a lot more "'of course length of hospital stay' correlates with 'had disease'" or "'Phone number' is a unique identifier in this data so of course it shows up most often in random forest predictor" moments.
Putting together weighted ensembles in R of a clean data set is basically manual labor but doing the same analysis at scale involves a lot of of nuances that should be considered data science. In particular you need to be able to determine if a problem can be split into mostly independent parallelizable sub problems (which also often requires domain knowledge) or reduced/relaxed into something that has a well established optimized solution (like matrix decomposition). And finally you need to be able to determine if your prototype and the production code are converging to the same results and debug it if it isn't which is non trivial with stochastic algorithms.
Re:Becoming a data scientist might be easier than you think
by rm999 on Monday October 15, 2012 @11:10AM
(#4655368)
> All the evidence from kaggle indicates that deep domain knowledge is not required
That's irrelevant, data scientists don't do data mining contests for a living. In my experience finding the right question to answer is a large chunk of data science, and that is never spelled out for you like it is in a contest.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @11:16AM
(#4655397)
You're saying kaggle is fundamentally different to what data scientists do? I don't understand that. The right question is usually how do I increase profits (or score this essay/image etc). So simply set up your problems that way.
Re:Becoming a data scientist might be easier than you think
by benhamner on Monday October 15, 2012 @12:54PM
(#4655927)
When we host competitions on Kaggle, a lot of work goes into asking the right question and structuring the problem. The domain expertise is incorporated in this step, as well as in putting the competition results to use in production.
This splits the "domain expertise" and "predictive modeling" components into two separate chunks. While domain expertise is crucial for asking the right questions, we've found that it isn't as necessary for the predictive modeling component. For example, in the essay scoring contest we hosted, none of the winners had touched natural language processing prior to the contest. However, they beat out many experts with decades of experience in NLP.
For an internal data science team, the "domain expertise" component is at least as important, as they are charged with asking the right questions as well. However, this does not mean competition winners cannot develop and learn this - they have already demonstrated their creativity and tenacity in one domain (applied machine learning), and this carries over nicely to other domains from our experience.
Re:Becoming a data scientist might be easier than you think
by rm999 on Monday October 15, 2012 @01:59PM
(#4656262)
>However, this does not mean competition winners cannot develop and learn this - they have already demonstrated their creativity and tenacity in one domain (applied machine learning), and this carries over nicely to other domains from our experience.
Strongly agreed. I actually got into the field through a company's data mining contest (pre-kaggle). I think people who are strong at building predictive models are great candidates for data sciences. But it took years of work experience after doing my graduate degree in machine learning to get to a point where I'm comfortable calling myself a decent data scientist.
It's easy to think model-building is the only important skill-set, but data and models don't exist in a vacuum; a more holistic view of where your data comes from and how your work will be used is essential. This excellent netflix blog entry illustrates what I'm saying quite well, I think. http://techblog.netflix.com/2012/04/netflix-recommendations-...
They make two points that illustrate the divide between a contest and the day-to-day work of a data scientist:
* The winning model was not usable in production. Netflix had to gut the 100+ model ensemble to a much simpler 2 model ensemble
* Business needs change, the question they were trying to answer changed from the start of the contest
Re:Becoming a data scientist might be easier than you think
by svasan on Monday October 15, 2012 @01:28PM
(#4656091)
I'd be curious to know if kaggle measures/publishes model performance, model degradation, etc., for the models that the companies/institutions ended up incorporating in their business activities.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @01:19PM
(#4656043)
You're actually from kaggle so I'm going to look like a troll arguing with you so I won't try (though I do privately think the right question is pretty obvious always - will this combination of parameters make profit - or some other obvious single metric. Just give all data to the data scientist and have them build the model).
Re:Becoming a data scientist might be easier than you think
by rm999 on Monday October 15, 2012 @11:35AM
(#4655491)
>You're saying kaggle is fundamentally different to what data scientists do
Yes. Your view of data science is extremely narrow, there is more to it than creating a predictive model to optimize a single metric. Reread the second paragraph of my first comment.
Re:Becoming a data scientist might be easier than you think
by Evbn on Monday October 15, 2012 @11:48AM
(#4655560)
Mostly yes, but "writing production grade software" is the distinction that separates the fast and loose ad hockey Data Science team from the Engineering team.
Re:Becoming a data scientist might be easier than you think
by dschiptsov on Monday October 15, 2012 @08:57AM
(#4654834)
Bad programming is easy. Idiots can learn it in 21 days..
I'm one of those who actually completed this course (with a score of 73.10/780) but it doesn't make you a data scientist. It is only the very beginning.
The course itself is a brilliant work of a passionate top-of-the-field professional. No wonder coursera.com is such a huge success.
Re:Becoming a data scientist might be easier than you think
by EzGraphs on Monday October 15, 2012 @09:34AM
(#4654950)
Yeah - particularly the mathematical/statistical side of things. But a big part of the profession is software development / computer science. An actuary might construct useful models and choose the proper techniques to analyze data, but choose inferior technologies or suboptimal implementations that won't work for large data sets.
A lot of actuaries spend most of their life inside of Excel. Excel won't cut it with Big Data.
Re:Becoming a data scientist might be easier than you think
by amalag on Monday October 15, 2012 @09:45AM
(#4654993)
I guess the professor isn't kidding when he says in the videos. "After you finish this course, you will know as much or more than the silicon valley programmers doing machine learning" The material he presents is quite distilled. He gives a lot of real examples, but the programming exercises are sort of fill in the blanks. A lot of the hard work is done. You can still learn a lot though.
Re:Becoming a data scientist might be easier than you think
by waterlesscloud on Monday October 15, 2012 @09:59AM
(#4655048)
This is a crucial step for the mooc's: students who have completed their classes and go on to real world achievements. It's precisely the way that a school builds a reputation, by the success of its students.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @10:22AM
(#4655152)
Which brings up the question of what all these academic machine learningists (especially the theorists) are up to? Why aren't they winning kaggle? Why aren't Andrew Ng's own grad students collecting the prizes? Some self-reflection needs to go on.
Re:Becoming a data scientist might be easier than you think
by imgabe on Monday October 15, 2012 @10:32AM
(#4655186)
Maybe they are spending their time conducting research instead of entering contests? I'm not trying to be snarky, but do you know that they're even entering the contests?
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @10:45AM
(#4655252)
Why are they not entering contests? What is the point of their research if it isn't to engineer better algorithms for the task of machine learning? How do they prove that their research led to better algorithms?
Re:Becoming a data scientist might be easier than you think
by imgabe on Monday October 15, 2012 @11:09AM
(#4655366)
I would imagine they publish papers that are reviewed by other academics. It is possible to use their algorithm on a dataset without entering a contest to do so.
It's a little like asking why Electrical Engineering PhDs are not designing the new iPhone (probably some are involved somewhere, I know). Research and applications are two separate endeavors.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @11:19AM
(#4655415)
The researchers actually do set up competitions to see who is best. However, there are few entrants. In kaggle there are 1000s of competitors, so it's a better judge of whether they are legitimately improving the state of the art (rather than simply schmoozing their local funding agencies).
Re:Becoming a data scientist might be easier than you think
by Evbn on Monday October 15, 2012 @11:52AM
(#4655597)
Oh, just noticed the username. I hope someone hires Marshall soon, or he finishes the book he is working on , or whatever, so this long form trolling can come to conclusion.
Re:Becoming a data scientist might be easier than you think
by bromang on Monday October 15, 2012 @11:12AM
(#4655377)
within the constraints of by the competition, the optimal method of winning is almost always to use well known existing algorithms and methods. academic research, however, is extremely time consuming and is of uncertain value ex ante. but without it we would not have many of the most popular machine learning tools used today...
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @11:44AM
(#4655538)
Well, capitalism, which is about fostering competition, has led to faster progress than the alternatives (assigning duties to specific people - communism or dictatorship).
Re:Becoming a data scientist might be easier than you think
by brainsareneat on Monday October 15, 2012 @12:26PM
(#4655760)
I don't understand how this is a response to the above point. Kaggle competitions and academic research are on a different timescale and have different goals. Most of the algorithms used by Kagglers were designed by academics - and those academics were rewarded for producing those algorithms with grants from both for-profit and non-profit organizations. Both individual and deeply collaborative endeavors have a place in capitalism and contribute massively to economic growth and scientific progress.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @12:44PM
(#4655859)
I understand what you're saying, but I'm skeptical that acdemic research is all that it's cracked up to be. Studies are always popping up about how most scientific findings are fraud. And then there's things like the X prizes and darpa challenges which really show who's right and who's wrong and what's actually possible. I'd much rather see open competitions used than grants. Kaggle can be that "central" competition for the machine learning community.
Re:Becoming a data scientist might be easier than you think
by imgabe on Monday October 15, 2012 @01:33PM
(#4656107)
But an open competition doesn't actually fund anything. The money is awarded after all of the work has been done. Somebody has to pay to feed and house scientists while they're working on something, as well as for lab equipment and support staff, etc, etc. An open competition solves none of those problems.
Furthermore, a competition only compensates the winner. That's all well and good for competitions, but sometimes science advances as much by finding things that don't work (and why they don't work) as by finding things that do.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @02:02PM
(#4656273)
There are probably solutions to all the above. "Tapered" prize giving (even 100th place gets a small prize), more frequent changeovers (rather than tenure and multiyear student commitments, have monthly or weekly reviews about wh stays and who leaves, just like in the private sector). Anyway, I acknowledge your points, I just think the present system isn't necessarily the perfect system.
Re:Becoming a data scientist might be easier than you think
by marshallp on Monday October 15, 2012 @12:00PM
(#4655637)
Are you sure grad students get paid more than that (that's who we're discussing here). And kaggle competitors aren't in it for the money. Almost every entrant will lose (it's winner take all) so all have them have day jobs.
Being a good data scientist requires a lot more than machine learning, including a solid understanding of the business side (deep domain knowledge), the ability to write production-grade software and tools, scripting/hacking/data munging, math/statistics, and common sense. Running a sanitized dataset through machine learning algorithms is maybe 5-10% of it.
I'm not trying to discourage people, I'm thrilled so many people are taking an interest in data sciences and I want to push interested people in a direction where they can excel at it. But this article is dangerous - becoming a data scientist requires a lot of hard work. I've seen a large sample of people (through interviews) who think a single online class is enough to get into the field. It's a great start, but if you want to be valuable you need a wider set of skills.
Does kaggle publish how the models performed under normal business conditions?
But you have to interpret the data within the context of the business need/requirement.
Building a credit risk model is vastly different from building a (personal) insolvency/bankruptcy model though both may entail the same set of steps in developing the model. The variables that make it to the model depend on the business need.
In kaggle, one of the datasets that I messed around with had variable labels as Var_1, Var_2, ..., Var_X. So while fitting a model, I would not know why a particular variable made it into the model. You can see that this kind of variable labeling does not give me any insight into how that variable was generated. I need to know whether the variable was raw/aggregated/transformed etc. And that takes you back to understanding the data in the context of the business/domain.
1) a key company policy 2) specific business activity 3) input coming from another model
It is better to
a) define the problem, b) collect the data, c) build the variable library, d) and then fit the model
rather than jump to step (d) directly because the modeler/scientist has greater understanding of the entire set of data going into the model development. It is very likely that the modeler would uncover any/all of the three influencing factors I mentioned above, during the data collection stage.
While kaggle is an interesting concept, from a different perspective it looks like an "effort harvesting" operation. For a pittance, the companies/institutions that are sponsoring the contests are getting a steal. (I am not sure if the million dollar prize is still up for grabs.) However, for folks who do want to break into data sciences/statistics field, kaggle certainly is a good platform to get acquainted with data science/statistics related skills.
For example, early analysis of a "fresh" data set involves a lot more "'of course length of hospital stay' correlates with 'had disease'" or "'Phone number' is a unique identifier in this data so of course it shows up most often in random forest predictor" moments.
Putting together weighted ensembles in R of a clean data set is basically manual labor but doing the same analysis at scale involves a lot of of nuances that should be considered data science. In particular you need to be able to determine if a problem can be split into mostly independent parallelizable sub problems (which also often requires domain knowledge) or reduced/relaxed into something that has a well established optimized solution (like matrix decomposition). And finally you need to be able to determine if your prototype and the production code are converging to the same results and debug it if it isn't which is non trivial with stochastic algorithms.
That's irrelevant, data scientists don't do data mining contests for a living. In my experience finding the right question to answer is a large chunk of data science, and that is never spelled out for you like it is in a contest.
This splits the "domain expertise" and "predictive modeling" components into two separate chunks. While domain expertise is crucial for asking the right questions, we've found that it isn't as necessary for the predictive modeling component. For example, in the essay scoring contest we hosted, none of the winners had touched natural language processing prior to the contest. However, they beat out many experts with decades of experience in NLP.
For an internal data science team, the "domain expertise" component is at least as important, as they are charged with asking the right questions as well. However, this does not mean competition winners cannot develop and learn this - they have already demonstrated their creativity and tenacity in one domain (applied machine learning), and this carries over nicely to other domains from our experience.
Strongly agreed. I actually got into the field through a company's data mining contest (pre-kaggle). I think people who are strong at building predictive models are great candidates for data sciences. But it took years of work experience after doing my graduate degree in machine learning to get to a point where I'm comfortable calling myself a decent data scientist.
It's easy to think model-building is the only important skill-set, but data and models don't exist in a vacuum; a more holistic view of where your data comes from and how your work will be used is essential. This excellent netflix blog entry illustrates what I'm saying quite well, I think. http://techblog.netflix.com/2012/04/netflix-recommendations-...
They make two points that illustrate the divide between a contest and the day-to-day work of a data scientist:
* The winning model was not usable in production. Netflix had to gut the 100+ model ensemble to a much simpler 2 model ensemble
* Business needs change, the question they were trying to answer changed from the start of the contest
edit - for clarity.
Yes. Your view of data science is extremely narrow, there is more to it than creating a predictive model to optimize a single metric. Reread the second paragraph of my first comment.
I'm one of those who actually completed this course (with a score of 73.10/780) but it doesn't make you a data scientist. It is only the very beginning.
The course itself is a brilliant work of a passionate top-of-the-field professional. No wonder coursera.com is such a huge success.
A lot of actuaries spend most of their life inside of Excel. Excel won't cut it with Big Data.
source: http://blog.kaggle.com/2012/10/04/engineering-practices-in-d...
Software development is important, but it's not everything. People choose what works and use it.
Though I agree that spreadsheets won't always work on Big Data, there's no point in doing more if you can achieve it with less.
It's a little like asking why Electrical Engineering PhDs are not designing the new iPhone (probably some are involved somewhere, I know). Research and applications are two separate endeavors.
Furthermore, a competition only compensates the winner. That's all well and good for competitions, but sometimes science advances as much by finding things that don't work (and why they don't work) as by finding things that do.