Re:Colly: Fast and Elegant Scraping Framework for Golang
by feelin_googley on Friday October 06, 2017 @02:36PM
(#15419208)
Improve the CSV:
# add to end
\
|exec tr '\15' '\12' \
|exec awk 'NR==1{a=$0}NR==3{b=$0}NR==5{print}NR>=6{print "\42"a"\42,"b","$0}'
# update CSV header line
\"sitetitle\",\"url\",\"pageID\",\"pagetitle1\",\"pagetitle2\",\"jsonurl\"/;};
Probably would be better to change "jsonurl" to "jsonfile" and just save the filename. The location (URL) of this file may be in several locations or it may change over time.
The ones they identify as "stunning" seem to be mostly devoid of text and are mainly images and videos.
As others in this thread have noted is common on today's web, they use json endpoints. Some sites also have an optional XML feed activated at /feed.xml.
Viewing the images and videos is possible without any use of Javascript. Simply extract the image and video urls and their descriptions from the json.
Below I only demonstrate how to extract the text, as html. One could view this html in a browser, without Javascript.
Random wix site: www.cricketcanadakids.com
Random page from the site: coach-profiles. JSON url taken from CSV above.