diggernaut
Re:Bypassing website anti-scraping protections
on Saturday June 02, 2018 @11:00AM (#17214849)
They recently removed User Agent and CSRF token from signature creation process. Right now used only rhx_gis parameter and URL decoded variables from query string to generate MD5 signature.
However, your findings about user agents looks interesting. I assume they may use user agent to generate rhx_gis. It could explain why auth doesnt work if you change single char in user agent.
Scraping business profiles from Instagram
New integration for Diggernaut: Zapier
How open data can change the world
Exporting data from Diggernaut to Tableau
How Are Things with Open Data in Russia
XLS, XLSX and CSV has been added as supported inbound format for our platform
Diggernaut and Death by Captcha = easy captcha bypass during web scraping process
How to scrape pages with infinite scroll: extracting data from Instagram
How to build a web scraper if your source is RSS feed
Augmenting JSON to XML Conversion in Golang
Diggernaut has got JSON Schema functionality implemented
Start to learn how Diggernaut's Excavator app works
Web scraping efficiency tests using different programming languages
Re:Ask HN: Rest api authorization
on Saturday October 15, 2016 @05:50PM (#12715811)
you can (and should :)) use it (token auth) with ssl. To get SSL is not a problem today, and will cost you nothing if you use letsencrypt.org
Re:Show HN: Diggernaut – Turn website content into data sets
on Saturday October 15, 2016 @04:22PM (#12715511)
Thank you guys for all these feedback for using website on mobile, we fixed index page, video should work now on phones, so as price list and also footer. And now fixing signup page for mobiles :)
Re:Show HN: Diggernaut – Turn website content into data sets
on Saturday October 15, 2016 @03:55PM (#12715424)
thank you for your opinion, we noted it. By the way you are paying for using cloud resources, not for using technology, you can compile digger, download it and run for free at your own server or computer. And you can do it with free plan. So Im not really sure why we are discussing cost of our subscription plans :)
We are in beta, so we would really appreciate some feedback about user experiece when they using website, visual extractor, or compiled diggers.
Legal side of web scraping is heavily depends on country you live in.
Re:Show HN: Diggernaut – Turn website content into data sets
on Saturday October 15, 2016 @01:35PM (#12714861)
Thanks a lot, we are working on a long list of changes to make make mobile users experience better, added your feedback to the list
Re:Show HN: Diggernaut – Turn website content into data sets
on Saturday October 15, 2016 @11:35AM (#12714321)
Thank you for your feedback, we will check and fix this issue shortly
Pushing data directly to RDBMS with Diggernaut
Show HN: Diggernaut – Turn website content into data sets
Re:Ask HN: Go or Python? Expanding my toolbelt
on Saturday October 15, 2016 @04:49AM (#12713285)
It actually depends on what you going to do with your knowledge, Go is definitely more efficient and faster than Python, on other hand Python is more popular, easier to use, has bigger set of available libraries.
Re:Ask HN: How to get started with machine learning?
on Saturday October 15, 2016 @04:37AM (#12713247)
You can start with free coursera course https://www.coursera.org/learn/machine-learning/, it starts 17 oct
and then continue with https://www.coursera.org/learn/neural-networks/