News for hackers. Stuff that matters.
Re:Developers are your users too – a defence of 'View Source' by polote on Saturday June 13, 2020 @05:40PM (#23512997)
Another issue, is that it is more and more difficult to create a web crawler as you need to render js to be sure to really see all content and you can't know beforehand if the pages needs Javascript rendering. As it cost much more to render js than parsing a html, only Google and a few other big Corps can afford it
Re:Developers are your users too – a defence of 'View Source' by walshemj on Saturday June 13, 2020 @06:12PM (#23513257)
And Google does it in two stages a first pass that's then handed off to a second stage for full rendering
Re:Developers are your users too – a defence of 'View Source' by ex3ndr on Saturday June 13, 2020 @07:54PM (#23513899)
Is it? we just have multiple instances of puppeteer that makes screenshots and dumps rendered HTML for parsing. Very simple service with autoscaling it is very affordable.
Re:Developers are your users too – a defence of 'View Source' by MereInterest on Saturday June 13, 2020 @10:08PM (#23514546)
Definitely more difficult than a single curl/wget request. You now need to parse the results, run a Turing-complete language, launch other requests based on output from that language, then wait for all results to be compiled into the final HTML, which may not even be possible for a continuously updated page.
Re:Developers are your users too – a defence of 'View Source' by ex3ndr on Sunday June 14, 2020 @02:45AM (#23515617)
That's all literally 10-20 lines of code and actually much shorter than downloading html manually.

Puppeteer can wait for final HTML.

Re:Developers are your users too – a defence of 'View Source' by MereInterest on Sunday June 14, 2020 @08:58AM (#23517357)
Solving the halting problem is now 10-20 lines of code? Because that would be needed in order to determine if a page is fully compiled to HTML.
Re:Developers are your users too – a defence of 'View Source' by ex3ndr on Sunday June 14, 2020 @01:49PM (#23519671)
This is not a problem, this feature is built in in Chrome and is literally one line: https://pptr.dev/#?product=Puppeteer&version=v3.3.0&show=api...