News for hackers. Stuff that matters.
Re:Colly: Fast and Elegant Scraping Framework for Golang by feelin_googley on Thursday October 05, 2017 @12:54PM (#15410654)
The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript.

Blogger has a feed. Here is one way to retrieve it, in two steps.

     1. get TargetBlogID
     2. retrieve data
     optional:
     3. format data for viewing

     Example:

     1.
     x=$(exec curl https://blogger.googleblog.com \
     |exec sed '
     s/\\046/\&/g;
     s/\\46/\&/g;
     s/\\075/=/g;
     s/\\75/=/g;
     /targetBlogID/!d;
     s/.*targetBlogID=//;
     s/&.*//;
     '); 

     2.
     curl -o y.htm https://www.blogger.com/feeds/$x/posts/default 

     3. 
     exec sed '
     # ^M is "\r"
     s/^[0-9a-f]*^M//; 
     s/&lt;/</g;
     s/&gt;/>/g;
     s/&amp;/\&/g;
     s/&quot;/\"/g;
     1i\
     <br><br>
     
     s/<name>/<br><br>name &/g;
     s/<uri>/<br>uri &/g;
     s/<generator>/<br>generator &/g;
     s/Blogger//;
     s/<id>/<br>id &/g;
     s/<published>/<br>published &/g;
     s/<email>/<br>email &/g;
     s/<title type=.text.>/<br><br>&/g;
     s/<openSearch:totalResults>/<br>total results &/g;
     s/<openSearch:startIndex>/<br>start index &/g;
     s/<openSearch:itemsPerPage>/<br>items per page &/g;
     s/<updated>/<br>updated &/g;
     s/<thr:total>/<br>thr:total &/g;
     s/<\/feed>/&<br><br><br>/;
     s/^M*/<br>/;
     ' y.htm \
     |exec tr -cd '\12\40-\176'
Re:Colly: Fast and Elegant Scraping Framework for Golang by always_good on Thursday October 05, 2017 @05:20PM (#15412673)
It's almost like you asked that question to bait someone into giving you an excuse to show "just how easy it is" with some cli hacking.
Re:Colly: Fast and Elegant Scraping Framework for Golang by dchuk on Thursday October 05, 2017 @05:42PM (#15412790)
Just 3 easy steps