Ask HN: Work to search for if I enjoy web scraping and browser automation
For a while now I’ve started to notice that what I really enjoy is the web scraping and browser automation I do for myself. I tend to use Ruby for the task, relying mostly on Nokogiri and Watir. I do it to automate repetitive tasks on websites or to build personal RSS/JSON feeds of websites that do not provide them (or are limited in what they provide). I don’t feel tied to those technologies. I enjoy them but feel confident I can learn another language/library enough to get the job done, if need be.
1. Having never worked professionally as a programmer, and having no formal training, what kind of job or contract/freelance work should I be searching for?
2. What’s the job title of someone that does this type of work?
3. Are there any specific skills I should be cultivating? For example, if most available work for these skills is in testing (is it?), should I be focusing on learning a testing framework?
4. Are there any specific technologies you’d recommend I learn? Why?
5. Any other advice?
Many QA departments need people who can automate their manual interaction step process to ensure that a service is behaving correctly.
It’s costly to manually check all the user stories that an application supports, and automating this can be a career path for an employee or a contractor.
My recommendation would be to get your foot in the door at a company that has a large manual QA process and is looking for someone to write for them automated tests, or convert their manual steps into tests.
1. For web scraping, you may get consulting jobs. For automation testing, search for job title of 'QA Automation Tester'.
2. Consultant / Tester / Analyst (Depending on seniority till you become Testing Manager)
3. I think there is a much larger market for testing. You can start by learning selenium. There are many testing frameworks, but it comes down to the company - which programming language and framework they use.
4. You can think of learning appium for mobile (android and iOS) automation.
5. Maybe you can try to search for work in Ruby for starts.
https://github.com/RealEstateWebTools/property_web_scraper
Take a look and let me know what you think. I have a few ideas about how to monetize. Get in touch if this is something you would like to work with me on ( ed at propertywebbuilder.com )
Might give you some ideas about markets, pricing etc
then create a portfolio
then start blogging.
reach out to other web dev/phone shops and mention what you specialize in and see if they come across this work and want to refer it to you or resell it, etc.
create social media profiles linking back to your site.