Re:Colly: Fast and Elegant Scraping Framework for Golang
by feelin_googley on Thursday October 05, 2017 @12:54PM
(#15410654)
The reason I asked is that I rarely use a graphical browser so I am always curious about wesbites that are inaccessible without Javascript.
Blogger has a feed. Here is one way to retrieve it, in two steps.
1. get TargetBlogID
2. retrieve data
optional:
3. format data for viewing
Example:
1.
x=$(exec curl https://blogger.googleblog.com \
|exec sed '
s/\\046/\&/g;
s/\\46/\&/g;
s/\\075/=/g;
s/\\75/=/g;
/targetBlogID/!d;
s/.*targetBlogID=//;
s/&.*//;
');
2.
curl -o y.htm https://www.blogger.com/feeds/$x/posts/default
3.
exec sed '
# ^M is "\r"
s/^[0-9a-f]*^M//;
s/</</g;
s/>/>/g;
s/&/\&/g;
s/"/\"/g;
1i\
<br><br>
s/<name>/<br><br>name &/g;
s/<uri>/<br>uri &/g;
s/<generator>/<br>generator &/g;
s/Blogger//;
s/<id>/<br>id &/g;
s/<published>/<br>published &/g;
s/<email>/<br>email &/g;
s/<title type=.text.>/<br><br>&/g;
s/<openSearch:totalResults>/<br>total results &/g;
s/<openSearch:startIndex>/<br>start index &/g;
s/<openSearch:itemsPerPage>/<br>items per page &/g;
s/<updated>/<br>updated &/g;
s/<thr:total>/<br>thr:total &/g;
s/<\/feed>/&<br><br><br>/;
s/^M*/<br>/;
' y.htm \
|exec tr -cd '\12\40-\176'
Re:Colly: Fast and Elegant Scraping Framework for Golang
by always_good on Thursday October 05, 2017 @05:20PM
(#15412673)
It's almost like you asked that question to bait someone into giving you an excuse to show "just how easy it is" with some cli hacking.
https://blogger.googleblog.com/
It's arguably a page of text not a web app that needs user interaction.
I'm not trying to start an argument about JS: the consensus on HN seems to be that if you don't execute JavaScript you don't deserve to read webpages. I'm just saying that your website's clients are more diverse than 'normal' well-sighted humans. There may be machines reading the site, for all kinds of reasons.
And regarding endpoints. One could make the argument that with AJAX we now have richer APIs. I disagree. We have a well-understood API for getting hypertext (HTTP) in a well-understood format (HTML) that works/worked across all websites. Replacing that with a custom-built API for every website isn't apples-for-apples.