Re:Web Scraping Indeed for Key Data Science Job Skills
by pc86 on Monday March 30, 2015 @01:39PM
(#9291129)
Love the design and my first query "developer" turned up some good results. Some feedback:
- If I enter "c#" as a query it simply refreshes the page.
- A lot of the "skills" I am getting back are simply rephrased job titles (e.g. "web developer" returned "web applications, web development, web services, mobile application development, support, responsibility, web design, javascript, project and software developer." for the skills list)
Definitely has a lot of promise though if you can reliable filter out skills from job descriptions.
It works by sampling job ads from Indeed, then applying some information extraction/retrieval/NLP algos to extract and weight the presence of identified skills and qualities. There's some occasional glitches in the algo (I need to fix some of the disambiguation data), but it usually gives reasonable results.
I was thinking of focusing the algorithm for giving really in-depth feedback on improving your resume specifically for a job. Now if only I could finish my PhD thesis I might actually have the time to do more with it...