News for hackers. Stuff that matters.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by AznHisoka on Thursday December 20, 2018 @11:09AM (#18726082)
This is why 90% of data science is the messy, grunt work of just simply cleaning/normalizing the data. The analysis is the easy part.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by screye on Thursday December 20, 2018 @11:50AM (#18726503)
> grunt work

Honestly, it is an insult to call good data preprocessing skills grunt work.

It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by minimaxir on Thursday December 20, 2018 @11:53AM (#18726530)
That's the point of the comment. Modern data science tutorials assume the source data is a nice and tidy CSV, when the reality is that data is very often a mess, and it's a constraint often not reflected in the glamorization of the field. (it's also a reason that I recommend that code always be published when making posts such as these, as it's very easy to do wrong!)
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by alexhutcheson on Thursday December 20, 2018 @12:03PM (#18726655)
Hadley Wickham's book "R for Data Science"[1] does a good job emphasizing the data cleaning and reshaping steps in the analysis process.

[1] https://r4ds.had.co.nz/

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by tfehring on Thursday December 20, 2018 @12:58PM (#18727234)
>Modern data science tutorials assume the source data is a nice and tidy CSV

For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science.

As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by minimaxir on Thursday December 20, 2018 @01:11PM (#18727379)
> Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data.

True, although ideally the data is cleaned/structured before it hits a data warehouse and the data scientist starts working with it. It's an iterative process (DS finds messy data, flags the problem upstream, repeat).

I do wish there were more tutorials with messy data; the ones I make I deliberately try to use complex datasets and highly relational ones, although as a result the tutorials are more complicated and more imposing for beginners.

> As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation.

When I was interviewing for DS jobs, I got a lot of the implement-binary-search-on-a-whiteboard questions which annoyed the hell out of me. But for the takehome assignments, they required some data ETL, which I felt was more representative. (some assignments had deliberately flawed datapoints the user was expected to identify and remove while not explicitly being told to do so)

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by mrleinad on Thursday December 20, 2018 @12:02PM (#18726644)
Grunt work noun [ U ] uk ​ /ˈɡrʌnt ˌwɜːk/ us ​ /ˈɡrʌnt ˌwɝːk/ informal. the basic, hard work, often physical or boring work, that is necessary for something to succeed.

They're just calling it like it is.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by scarejunba on Thursday December 20, 2018 @04:32PM (#18729329)
My God. This "it is an insult" thing is insufferable. Please stop doing it.

It's literally what the phrase means.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by jMyles on Thursday December 20, 2018 @12:12PM (#18726770)
It's insulting to insinuate that grunt work is insulting.

(lightheartedly; I'm not like disgusted over here :-) )

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by donaldknuth123 on Thursday December 20, 2018 @01:52PM (#18727813)
Eventually it will be split from actual Data Science work which is essentially either Machine Learning or Statistics. Imagine making a master sculptor mine his own Marble, it would be absolutely absurd similarly it would be silly to make a PhD Machine Learning expert work on data cleaning. A massive waste of talent on trivial work.

Edit: trivial isn't the right word, it's obviously an important step in the process but quite different from what you are really paying a "Data Scientist/ML Engineer/etc" to do.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by jmalicki on Thursday December 20, 2018 @05:36PM (#18729829)
It's quite different... I would hope the marble miner would know quite a bit about geology, mining equipment, etc....

It's not that putting the sculptor on the task would be a waste, nearly as much as that the sculptor doesn't possess any of the necessary skills.

How many people know how to find a source of marble, run a mine, and sculpt a great work of art? Roughly nil, but it's how many organizations seem to hire.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by xapata on Thursday December 20, 2018 @10:06PM (#18731159)
On the contrary, if the statisticians aren't intimately familiar with the data and the cleaning process, it's likely that they'll overlook some important features. When I interview potential hires, I make sure they'll be self-sufficient when writing data extraction code.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by traek on Thursday December 20, 2018 @04:35PM (#18729362)
This is why most large tech companies have data engineer roles that are separate from the data science role.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by rhizome on Thursday December 20, 2018 @03:33PM (#18728854)
I worked for a company, which is still in business, 80% of whose work involves "cleaning/normalizing" data, basically everything that isn't sales and client services.

Market research is all about getting different kinds of data and massaging it into queryable and productizable form (reports).

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by AznHisoka on Thursday December 20, 2018 @04:53PM (#18729495)
Where do they usually get the market research data from?
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by rhizome on Saturday December 22, 2018 @01:08AM (#18739232)
Depends on the market being researched. Focus groups are one source, POS data is another. Large operators like Nielsen collect all kinds of data for all kinds of industries, which can be purchased. You collect what exists where it lies for the stuff you want to look at.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by z3t4 on Thursday December 20, 2018 @03:54PM (#18729038)
Often it can be faster to enter data manually then to make a script collect it automatically with the same accuracy.