Show HN: Web scraping page analyzer

Re:Show HN: Web scraping page analyzer by jardah on Tuesday February 13, 2018 @01:11PM (#16369299)

This tool basicaly performs the simplest data loading, it opens the webpage, then waits till most xhr requests are done, wait's a second (tio give JS time to manipulate DOM) and then loads data from the page. This way, it has what user sees when he opens the page in browser. So if the data is visible, or loaded through XHR or hidden in global JS variable it will see it.

For more advanced usage (like clicking, or submiting a search request) it would need to have some kind of scenario like: "Click on this" -> "wait till this loads" -> "type something here" -> "scroll to this" -> load data.

Which is possible with headless chrome, so the trick is to make it general and easy to use (something like recording what user does through chrome plugin). Maybe in future versions :)

[ Parent | on HN ]

Re:Show HN: Web scraping page analyzer by cseelus on Tuesday February 13, 2018 @02:49PM (#16370114)

Could be an interesting enhancement. Sounds a little bit like what Capybara, a test framework for Ruby apps can do[1], things like

  click_link('Link Text')
  fill_in('Password', with: 'Seekrit')
  choose('A Radio Button')
  check('A Checkbox')
  uncheck('Another Checkbox')
  select('Option', from: 'Select Box')

1) https://github.com/teamcapybara/capybara#navigating

[ Parent | on HN ]