News for hackers. Stuff that matters.

A comment on Data Science Is America’s Hottest Job. Parent comment

Re:Data Science Is America’s Hottest Job by make3 on Friday May 18, 2018 @11:58AM (#17102478)
In real life, problems are hard and require theoretical investigation to do well.

That's the actual job! The theoretical investigation. Not the programming. The programming is really easy. which is part of why having 15 years in software doesn't have so much weight.

[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by maxxxxx on Friday May 18, 2018 @12:06PM (#17102533)
How many data scientists really do theoretical investigation? From what I see not too many. It feels a little like looking only for devs who can write complete operating systems and then having them write CRUD apps.
[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by superhuzza on Friday May 18, 2018 @04:34PM (#17104746)
That's basically the job - people who can get, transform and interpret the data correctly.
[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by gaius on Friday May 18, 2018 @12:14PM (#17102602)
90% of data science time is spent sourcing and then cleaning the data. 10% is spent preparing and delivering reports to stakeholders. The ML time is a rounding error.

The job in which you are magically given a pristine dataset and can just tweak ML models all day doesn’t exist.

[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by Puer on Friday May 18, 2018 @12:21PM (#17102656)
The running assumption on HN seems to be that ML/data science jobs are easy because you can just "plug in" already implemented algorithms and just expect things to work. I think the barrier for most of these jobs is a proper statistics education--one that involves active exploration of real world data that's imperfect and has to be cleaned. Very rarely is there an obvious solution to big data problems and much of the work is tedious parameter tuning that absolutely does require knowledge of the math and domain and not just how to write k-Means.
[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by gaius on Saturday May 19, 2018 @04:04AM (#17107081)
I think the barrier for most of these jobs is a proper statistics education--one that involves active exploration of real world data that's imperfect and has to be cleaned.

It's a bit of both really. If the data science job is in the field of predictive maintenance for example, then a (relatively) simple model may be sufficient to add business value straight away, and the hard part is a deep understanding of the potential failure modes of the machinery you're predicting on and the kinds of sensors used to gather the raw data. There's no "one size fits all".

This is one of my favourite papers on the subject, written by one of the lecturers on the DS course I did: https://users.cs.duke.edu/~cynthia/docs/RudinETAL2010.pdf

[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by mmsimanga on Friday May 18, 2018 @01:37PM (#17103336)
This 1000 times. I work in data warehousing and Business Intelligence. The closest I have gotten to data scientists are actuaries. Most who don't have a clue about data management, the process to get your data to be consistent and clean so you can do your analysis. Instead of creating standardised processes and tables, each analysis is started from scratch leading to maintenance nightmare and inability to cross-reference the results with any other data sets.

Whoever is running data scientist units needs to realise you need IT/DBA/programming people in the mix. Statisticians in my experience cannot design databases, concepts of code reuse and normalisation do not exist in their vocabulary. A great deal of time is wasted doing repetitive tasks that someone with a programming background would have assisted.

[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by make3 on Friday May 18, 2018 @04:41PM (#17104790)
If you're a programmer by formation, then indeed you will be given a programming intensive, theoretically non-intensive task (or be found to be qualified for such a job). If you are more theoretically inclined, unsurprisingly, you end up with a more theoretically heavy job.
[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by PunchTornado on Friday May 18, 2018 @12:06PM (#17102527)
I would say the non-programming part, the theoretical investigation is really easy. Plus the majority of data science do data cleaning 80% of their time.
[ Parent | on HN ]
Re:Data Science Is America’s Hottest Job by make3 on Friday May 18, 2018 @04:42PM (#17104797)
If you're a programmer by formation, then indeed you will be given a programming intensive, theoretically non-intensive task (or be found to be qualified for such a job). If you are more theoretically inclined, unsurprisingly, you end up with a more theoretically heavy job.
[ Parent | on HN ]