News for hackers. Stuff that matters.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by OisinMoran on Thursday December 20, 2018 @10:30AM (#18725694)
I think there are some flaws in this analysis that can be confirmed when the code is released.

I think the reason Scala seems surprisingly popular is that is was simple string matching so a post that said "We are looking for someone who can build scalable infrastructure" would count as one mention of "scala".

Could be wrong though.

Also see: https://news.ycombinator.com/reply?id=18725655&goto=item%3Fi...

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by AznHisoka on Thursday December 20, 2018 @11:09AM (#18726082)
This is why 90% of data science is the messy, grunt work of just simply cleaning/normalizing the data. The analysis is the easy part.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by screye on Thursday December 20, 2018 @11:50AM (#18726503)
> grunt work

Honestly, it is an insult to call good data preprocessing skills grunt work.

It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by minimaxir on Thursday December 20, 2018 @11:53AM (#18726530)
That's the point of the comment. Modern data science tutorials assume the source data is a nice and tidy CSV, when the reality is that data is very often a mess, and it's a constraint often not reflected in the glamorization of the field. (it's also a reason that I recommend that code always be published when making posts such as these, as it's very easy to do wrong!)
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by alexhutcheson on Thursday December 20, 2018 @12:03PM (#18726655)
Hadley Wickham's book "R for Data Science"[1] does a good job emphasizing the data cleaning and reshaping steps in the analysis process.

[1] https://r4ds.had.co.nz/

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by tfehring on Thursday December 20, 2018 @12:58PM (#18727234)
>Modern data science tutorials assume the source data is a nice and tidy CSV

For what it's worth, I don't really see this as a problem in and of itself. Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data. But it's good that tutorials segment these out, since not everyone needs to learn both at the same time. Anecdotally, I had plenty of real-world experience dealing with messy data by the time I started learning data science.

As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation. Granted, they're still really time-consuming as it is, but there's probably some room for optimization there.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by minimaxir on Thursday December 20, 2018 @01:11PM (#18727379)
> Of course, if you want to do data science in the real world, you need to learn about data cleaning, manipulation, and warehousing in addition to the "pure" data science process that begins with tidy data.

True, although ideally the data is cleaned/structured before it hits a data warehouse and the data scientist starts working with it. It's an iterative process (DS finds messy data, flags the problem upstream, repeat).

I do wish there were more tutorials with messy data; the ones I make I deliberately try to use complex datasets and highly relational ones, although as a result the tutorials are more complicated and more imposing for beginners.

> As an aside, my impression is that technical assessments for data science positions tend to underemphasize data cleaning and manipulation.

When I was interviewing for DS jobs, I got a lot of the implement-binary-search-on-a-whiteboard questions which annoyed the hell out of me. But for the takehome assignments, they required some data ETL, which I felt was more representative. (some assignments had deliberately flawed datapoints the user was expected to identify and remove while not explicitly being told to do so)

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by mrleinad on Thursday December 20, 2018 @12:02PM (#18726644)
Grunt work noun [ U ] uk ​ /ˈɡrʌnt ˌwɜːk/ us ​ /ˈɡrʌnt ˌwɝːk/ informal. the basic, hard work, often physical or boring work, that is necessary for something to succeed.

They're just calling it like it is.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by scarejunba on Thursday December 20, 2018 @04:32PM (#18729329)
My God. This "it is an insult" thing is insufferable. Please stop doing it.

It's literally what the phrase means.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by jMyles on Thursday December 20, 2018 @12:12PM (#18726770)
It's insulting to insinuate that grunt work is insulting.

(lightheartedly; I'm not like disgusted over here :-) )

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by donaldknuth123 on Thursday December 20, 2018 @01:52PM (#18727813)
Eventually it will be split from actual Data Science work which is essentially either Machine Learning or Statistics. Imagine making a master sculptor mine his own Marble, it would be absolutely absurd similarly it would be silly to make a PhD Machine Learning expert work on data cleaning. A massive waste of talent on trivial work.

Edit: trivial isn't the right word, it's obviously an important step in the process but quite different from what you are really paying a "Data Scientist/ML Engineer/etc" to do.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by jmalicki on Thursday December 20, 2018 @05:36PM (#18729829)
It's quite different... I would hope the marble miner would know quite a bit about geology, mining equipment, etc....

It's not that putting the sculptor on the task would be a waste, nearly as much as that the sculptor doesn't possess any of the necessary skills.

How many people know how to find a source of marble, run a mine, and sculpt a great work of art? Roughly nil, but it's how many organizations seem to hire.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by xapata on Thursday December 20, 2018 @10:06PM (#18731159)
On the contrary, if the statisticians aren't intimately familiar with the data and the cleaning process, it's likely that they'll overlook some important features. When I interview potential hires, I make sure they'll be self-sufficient when writing data extraction code.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by traek on Thursday December 20, 2018 @04:35PM (#18729362)
This is why most large tech companies have data engineer roles that are separate from the data science role.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by rhizome on Thursday December 20, 2018 @03:33PM (#18728854)
I worked for a company, which is still in business, 80% of whose work involves "cleaning/normalizing" data, basically everything that isn't sales and client services.

Market research is all about getting different kinds of data and massaging it into queryable and productizable form (reports).

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by AznHisoka on Thursday December 20, 2018 @04:53PM (#18729495)
Where do they usually get the market research data from?
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by rhizome on Saturday December 22, 2018 @01:08AM (#18739232)
Depends on the market being researched. Focus groups are one source, POS data is another. Large operators like Nielsen collect all kinds of data for all kinds of industries, which can be purchased. You collect what exists where it lies for the stuff you want to look at.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by z3t4 on Thursday December 20, 2018 @03:54PM (#18729038)
Often it can be faster to enter data manually then to make a script collect it automatically with the same accuracy.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by chefandy on Thursday December 20, 2018 @10:54AM (#18725936)
VaLUAble. tRUST.

I would be pretty surprised to see these two so much more commonly requested than Javascript.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by mchaver on Thursday December 20, 2018 @10:40AM (#18725789)
Probably the same for rust, looking at the October 2018 data set with https://kennytilton.github.io/whoishiring/ rust gives 50 results, trust gives 21, \brust gives 26. Similar results for scala.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by JshWright on Thursday December 20, 2018 @10:35AM (#18725737)
Yeah, similarly with "Excel". Are there really that many posts looking for people with deep spreadsheet experience? Seems more likely that it's a lot of false positives for "excellent" and "excel at".
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by osoba on Thursday December 20, 2018 @10:47AM (#18725862)
Excel is probably mislabelled, it should say HTML instead. Look at the list above
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by fredbo22 on Thursday December 20, 2018 @11:11AM (#18726108)
Yup. JavaScript, Node.js, Node, npm, React, Angular, Vue, Express and a few more words mean pretty the same thing and are fishing for the same pool of developers (Yes, I know there's a difference between Angular and React.)
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by jenscow on Thursday December 20, 2018 @11:33AM (#18726318)
> Yes, I know there's a difference between Angular and React.

When can you start?

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by adrianmonk on Thursday December 20, 2018 @02:07PM (#18727967)
I've noticed that in the hiring threads, a particular company is always hiring, and the email contact username is "austin". It would be easy to pick this up as a location even though it's a person's name.

This analysis has the Austin as one of the top 10 locations, and I'm curious whether or not that is skewing the numbers. There are a steady stream of jobs actually in Austin, so either seems possible.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by NorthOf33rd on Thursday December 20, 2018 @01:14PM (#18727427)
Same with product manager. Many descriptions list working with product management as a requirement. There have not been 425 product management jobs listed this year. Trust me, I've applied to most of them.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by lettergram on Thursday December 20, 2018 @11:35AM (#18726334)
You’re very likely correct. See the compare by volume:

https://hnprofile.com/compare?search=scala,python,rust

Not the only issue I see, for instance my company posted for “machine learning engineers”, yet that isn’t in the results at all.

I think they are doing regex search in the strings, as opposed to identifying the words and doing comparisons

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by fulafel on Friday December 21, 2018 @03:57AM (#18732164)
Here's a link with more languages in case anyone else got curious: https://hnprofile.com/compare?search=scala,python,rust,cloju...
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by ashelmire on Thursday December 20, 2018 @11:40AM (#18726392)
Gotta add some word boundaries to that regex!
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by NicoJuicy on Thursday December 20, 2018 @12:11PM (#18726759)
I wonder what's included for the regex of .net. Eg: asp.net core, dotnet, c#, VB.net, f#, .net, xamarin,...

Because there are a multitude of possibilities.

Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by emmanueloga_ on Thursday December 20, 2018 @08:00PM (#18730666)
Agreed. It looks like the author may be looking for something like a /.net/ regexp? I don't think .net is popular at all in HN job postings. A better regex would be pretty much what you suggest:

    /asp\.net|dotnet|c#|VB.net|f#|xamarin/i
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by danieka on Thursday December 20, 2018 @02:27PM (#18728195)
It appears that the same problem occurs with locations. For example it would appear that the small town of Visby (Sweden) which is 1/100 of the size of Stockholm has 1/10 the jobs of Stockholm. Or more likely the location matches the company name Visby of startup visby.io.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by TheAdamist on Thursday December 20, 2018 @02:59PM (#18728562)
Pittsburgh (120) seems to be beating Pittsburg (6) as well, so theres a lot of data cleanup to be done.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by rhizome on Thursday December 20, 2018 @03:36PM (#18728884)
I think there's also a bug where the numbers don't account for "among those who mentioned [category]."
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by alehul on Thursday December 20, 2018 @10:51AM (#18725904)
Would there be an issue with just changing the string to be matched to "scala " or " rust"?
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by meritt on Thursday December 20, 2018 @10:55AM (#18725954)
There are numerous ways to fix the glaring flaws in this analysis but a very simple approach would be using regex word boundaries, that way it'll work not only for the space but for commas, periods, end-of-line, etc.
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by jgtrosh on Thursday December 20, 2018 @10:54AM (#18725935)
The correct method is probably something like "\<$lang\>"
Re:Show HN: Analysis of 2018 Hacker News “Who Is Hiring” Job... by PhasmaFelis on Thursday December 20, 2018 @02:42PM (#18728378)
For the record, the original post has been edited: your suspicion was correct, and they've fixed it now.