News for hackers. Stuff that matters.
Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Thursday October 05, 2017 @11:46AM (#15410093)
"More and more webpages are requiring JS even to display anything beyond a blank page."

Can you provide some example webpages so we can take a look?

Also, I agree with @asciimoo's point about endpoints. One could make an argument that compared to the website design of the 1990's and 2000's, retrieving structured data from websites is actually getting easier not more difficult. I recall one period of time where the trend was to design websites entirely in Macromedia/Adobe Flash.

Here is what is missing from this project (and many others like it): when providing software that performs text processing one needs to not only provide example code but also example output. This enables a user to quickly compare her current text-processing solution with the software being provided without having to install, review and run the unfamiliar software.

For example, without some sample output she cannot test, e.g., whether her current solution produces the same output faster or using less code.

Re:Colly: Fast and Elegant Scraping Framework for Golang by afandian on Thursday October 05, 2017 @12:20PM (#15410374)
With a minute's searching, here's a webpage that's meaningless without JavaScript:

https://blogger.googleblog.com/

It's arguably a page of text not a web app that needs user interaction.

I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons.

And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.

Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Thursday October 05, 2017 @12:54PM (#15410654)
The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript.

Blogger has a feed. Here is one way to retrieve it, in two steps.

     1. get TargetBlogID
     2. retrieve data
     optional:
     3. format data for viewing

     Example:

     1.
     x=$(exec curl https://blogger.googleblog.com \
     |exec sed '
     s/\\046/\&/g;
     s/\\46/\&/g;
     s/\\075/=/g;
     s/\\75/=/g;
     /targetBlogID/!d;
     s/.*targetBlogID=//;
     s/&.*//;
     '); 

     2.
     curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 

     3. 
     exec sed '
     # ^M is "\r"
     s/^[0-9a-f]*^M//; 
     s/&lt;/</g;
     s/&gt;/>/g;
     s/&amp;/\&/g;
     s/&quot;/\"/g;
     1i\
     <br><br>
     
     s/<name>/<br><br>name &/g;
     s/<uri>/<br>uri &/g;
     s/<generator>/<br>generator &/g;
     s/Blogger//;
     s/<id>/<br>id &/g;
     s/<published>/<br>published &/g;
     s/<email>/<br>email &/g;
     s/<title type=.text.>/<br><br>&/g;
     s/<openSearch:totalResults>/<br>total results &/g;
     s/<openSearch:startIndex>/<br>start index &/g;
     s/<openSearch:itemsPerPage>/<br>items per page &/g;
     s/<updated>/<br>updated &/g;
     s/<thr:total>/<br>thr:total &/g;
     s/<\/feed>/&<br><br><br>/;
     s/^M*/<br>/;
     ' y.htm \
     |exec tr -cd '\12\40-\176'
Re:Colly: Fast and Elegant Scraping Framework for Golang by always_good on Thursday October 05, 2017 @05:20PM (#15412673)
It's almost like you asked that question to bait someone into giving you an excuse to show "just how easy it is" with some cli hacking.
Re:Colly: Fast and Elegant Scraping Framework for Golang by dchuk on Thursday October 05, 2017 @05:42PM (#15412790)
Just 3 easy steps
Re:Colly: Fast and Elegant Scraping Framework for Golang by Animats on Thursday October 05, 2017 @03:54PM (#15412106)
Wix. Everything with Wix.[1]

[1] http://www.wix.com/stunningwebsites/400

Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Friday October 06, 2017 @01:50AM (#15414708)
I looked at some exemplary Wix sites since I rarely ever encounter these.

The ones they identify as "stunning" seem to be mostly devoid of text and are mainly images and videos.

As others in this thread have noted is common on today's web, they use json endpoints. Some sites also have an optional XML feed activated at /feed.xml.

Viewing the images and videos is possible without any use of Javascript. Simply extract the image and video urls and their descriptions from the json.

Below I only demonstrate how to extract the text, as html. One could view this html in a browser, without Javascript.

Random wix site: www.cricketcanadakids.com

     curl -o x.htm https://www.cricketcanadakids.com

     # produce a listing of the page titles and json endpoint urls
     # in addition to a title, each page has a "pageID"
     # beginning on 3rd line is CSV e.g. for database: 
     # ID, title1, title2, url
     # last line of CSV is "master" json
     x=https://static.wixstatic.com/sites/; 
     exec awk '{gsub(/:{/,/"\n"{/);print}' \
     |exec sed -n '
      s/{/\
      &/g;
      s/\],/\
      &/;
      /<title>/p;
      /mainPage/p;
     ' \
     |exec sed '
     /var publicModel/d;
     /.*\"baseUrl\"/d;
     /{filename}/d;
     s/<title>//;s/<\/title>//;
     /ExternalBaseUrl/{s/.*Url\":\"//;s/\".*/\
     \
     \"ID\",\"title1\",\"title2\",\"url\"/;};

     s/\\//g;
     s/{\"pageId\":\"//;
     3!s/\",\"/,/g;
     s/title\"://;
     s/,pageUriSEO//;
     s/:\"/,/;
     s>pageJsonFileName\":\">'"$x"'>;
     s/\"}.*/.z/;
     s/\],\"mainPageId\",//;
     s>masterPageJsonFileName\":\">master,master,'"$x"'>;
     /topology\":\[/s/json.*/json.z/;
     ' x.htm
Random page from the site: coach-profiles. JSON url taken from CSV above.

     curl -4o y.json https://static.wixstatic.com/sites/7f1cbe_7131eb80aa297a10c02a08c8ffbc3ef6_122.json.z

     # produce simple html version of page
     # text-only, no images or videos
     x=$(exec echo b|exec tr b '\34');
     exec tr -cd '\12\40-\176' < y.json \
     |exec sed 's/{/'"$x"'&/g;s/ *//;' \
     |exec tr -d '\12' \
     |exec tr  '\34' '\12' \
     |exec sed '1d;$d' \
     |exec sed '
     s/</\
     &/g;
     ' \
     |exec sed 's/\\["tn]//g;s/\",\".*//;/{/d;'
Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Friday October 06, 2017 @02:36PM (#15419208)
Improve the CSV:

   # add to end
   \
   |exec tr '\15' '\12' \
   |exec awk 'NR==1{a=$0}NR==3{b=$0}NR==5{print}NR>=6{print "\42"a"\42,"b","$0}' 

   # update CSV header line 
     \"sitetitle\",\"url\",\"pageID\",\"pagetitle1\",\"pagetitle2\",\"jsonurl\"/;};
Probably would be better to change "jsonurl" to "jsonfile" and just save the filename. The location (URL) of this file may be in several locations or it may change over time.
Re:Colly: Fast and Elegant Scraping Framework for Golang by dithering on Thursday October 05, 2017 @12:46PM (#15410600)
I know of a site where all the data is embedded inline ("var Person = {firstName: 'Fred', lastName: 'Smith' }") then rendered to HTML with JS. No endpoints, Javascript not JSON, complete pain in the ass to extract without rendering in a headless browser first.
Re:Colly: Fast and Elegant Scraping Framework for Golang by sitepodmatt on Friday October 06, 2017 @12:03AM (#15414336)
Isn't this better? Why do you need it render when they're giving you basically JSON of the data (might need to quote the keys in strict parsers), just extract that.
Re:Colly: Fast and Elegant Scraping Framework for Golang by dithering on Friday October 06, 2017 @06:57AM (#15415773)
The short answer is that, yes, the only parser available to me is that strict. If I'm going to jump languages, I might as well go the whole hog and jump to Puppeteer.
Re:Colly: Fast and Elegant Scraping Framework for Golang by faitswulff on Thursday October 05, 2017 @12:23PM (#15410393)
Feedly, Netflix, Google Maps, as of this article:

https://sonniesedge.co.uk/blog/a-day-without-javascript

Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Thursday October 05, 2017 @01:08PM (#15410741)
I remember that article.

I do not use any of those three websites so I would be curious to know some specific examples of what the desired structured data looks like.

If for Netflix the desired data is only DVD summaries, I recall that these were easily retrievable without Javascript not too long ago.

Re:Colly: Fast and Elegant Scraping Framework for Golang by afandian on Thursday October 05, 2017 @01:46PM (#15411014)
I would argue that it's reasonable for these examples to require JS. My complaint above was the accessibility text-only sites.
Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Thursday October 05, 2017 @01:58PM (#15411128)
I would agree. That article had a very poor choice of websites, IMO. The web is quite usable without Javascript IME.
Re:Colly: Fast and Elegant Scraping Framework for Golang by slezyr on Thursday October 05, 2017 @12:56PM (#15410671)
> Can you provide some example webpages so we can take a look?

learnopengl.com

Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Thursday October 05, 2017 @01:55PM (#15411092)
curl -d 'content_name=Heading' https://example.com/content_load.php?

curl -d 'content_name=Heading/Subheading' https://example.com/content_load.php?

Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Friday October 06, 2017 @01:54AM (#15414726)

  s/example/learnopengl/
  s/Heading/Introduction/