Re:Show HN: I built an auto aggregating bot, collecting tren...
by rezashirazian on Monday September 19, 2016 @04:10PM
(#12534299)
Because many people hotlink directly from the site. And If I'm going to host the images myself and eat the cost of bandwidth I might as well derive some value from it.
I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.
Re:Show HN: I built an auto aggregating bot, collecting tren...
by rezashirazian on Friday September 16, 2016 @02:27PM
(#12516153)
I aggregate from Buzzfeed and 9gag as well, although I don't have them actively running.
I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself)
I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.
Re:Show HN: I built an auto aggregating bot, collecting tren...
by rezashirazian on Friday September 16, 2016 @02:01AM
(#12512151)
Ruby on Rails for the backend, Angular on the front. It's hosted on Heroku, it uses cloudfront, S3 for storage, Postgres for DB along with httpClient and mechanize for crawling and Paperclip for image manipulation.
Re:Show HN: I built an auto aggregating bot, collecting tren...
by rezashirazian on Friday September 16, 2016 @11:03AM
(#12514476)
Not particularly. The one thing that helps but most people don't know about reddit is the fact that adding a .json to the end of each url displays the content of that page in json format.
for example:
reddit.com/r/funny.json
This make crawling/fetching content from reddit much more trivial than old school web crawling.
Re:Show HN: I built an auto aggregating bot, collecting tren...
by rezashirazian on Monday September 19, 2016 @04:13PM
(#12534314)
I have written a crawler for buzzfeed and 9gag but I don't have them running because there is a lot of overlap between 9gag and reddit and they require more work.
To be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.
I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.
I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself)
I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.
for example: reddit.com/r/funny.json
This make crawling/fetching content from reddit much more trivial than old school web crawling.
To be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.