News for hackers. Stuff that matters.
Re:Show HN: ParseHub – Extract data from dynamic websites by gondo on Tuesday September 23, 2014 @03:13PM (#8357499)
can you be more specific please? as far as i know, you can create chrome extension what executes custom javascript code on the current page and pass the results to extension. so what exactly is the problem?
Re:Show HN: ParseHub – Extract data from dynamic websites by tsergiu on Tuesday September 23, 2014 @03:29PM (#8357617)
Sure, I'll give you one example.

We want to show a sample immediately as a user changes what they extract. On a static website, this is fairly easy. You simply run what the user created on the currently visible page.

However, when you involve interactivity, you can no longer do that. The major problem is idempotent operations. Imagine a click that changed the dom of a page. And now imagine running the sample on that same page. Re-running the sample may no longer work, because the click could have changed the page in such a way that the extraction no longer works (e.g. it deletes an element from the page).

To solve this issue, we actually reset a "hidden tab" to the starting state of the page you're on. This happens every time you re-run a sample. Unfortunately, it's not possible with Chrome to create such hidden tabs. We also mess with the cache to make sure that this tab can be reset really quickly, something that we couldn't find an API for with chrome.

Hope that answers your question.

Re:Show HN: ParseHub – Extract data from dynamic websites by tilt on Wednesday September 24, 2014 @03:21AM (#8360275)
Did you have a look at https://developer.chrome.com/extensions/background_pages ?

Not sure if it does fit all your needs.

Re:Show HN: ParseHub – Extract data from dynamic websites by totallynotcool on Tuesday September 23, 2014 @04:35PM (#8358028)
Could you clone the DOM into a virtual HTML element and save/serialize that?
Re:Show HN: ParseHub – Extract data from dynamic websites by tsergiu on Tuesday September 23, 2014 @06:19PM (#8358700)
That doesn't work because you need the javascript as well.
Re:Show HN: ParseHub – Extract data from dynamic websites by gondo on Tuesday September 23, 2014 @03:55PM (#8357745)
i see. thanks for taking time to explain.