News for hackers. Stuff that matters.
Re:Show HN: I built an auto aggregating bot, collecting tren... by wingerlang on Saturday September 17, 2016 @07:40AM (#12520108)
Why are you watermarking the images?
Re:Show HN: I built an auto aggregating bot, collecting tren... by rezashirazian on Monday September 19, 2016 @04:10PM (#12534299)
Because many people hotlink directly from the site. And If I'm going to host the images myself and eat the cost of bandwidth I might as well derive some value from it.

I provide a link to where the original images was taken from, so if anyone is interested in the image without the watermark they can get it from there.

Re:Show HN: I built an auto aggregating bot, collecting tren... by 0xfff on Friday September 16, 2016 @06:53AM (#12513096)
Is this just a reddit scraper or am I missing something?
Re:Show HN: I built an auto aggregating bot, collecting tren... by rezashirazian on Friday September 16, 2016 @02:27PM (#12516153)
I aggregate from Buzzfeed and 9gag as well, although I don't have them actively running.

I try to give the images a clutter-free, mobile friendly, reliable, fast and ad-free home. (I compress and host all the images myself)

I try to provide an easy mechanism for navigating to previous snapshots of what was aggregated for anyone who is passively browsing and has viewed all the recent images.

Re:Show HN: I built an auto aggregating bot, collecting tren... by ksashikumar on Thursday September 15, 2016 @11:57PM (#12511729)
How did you do this? What tech stack do you use?
Re:Show HN: I built an auto aggregating bot, collecting tren... by rezashirazian on Friday September 16, 2016 @02:01AM (#12512151)
Ruby on Rails for the backend, Angular on the front. It's hosted on Heroku, it uses cloudfront, S3 for storage, Postgres for DB along with httpClient and mechanize for crawling and Paperclip for image manipulation.
Re:Show HN: I built an auto aggregating bot, collecting tren... by ksashikumar on Friday September 16, 2016 @03:36AM (#12512488)
Do you use any kinda algorithm for crawling/fetching content from reddit?
Re:Show HN: I built an auto aggregating bot, collecting tren... by rezashirazian on Friday September 16, 2016 @11:03AM (#12514476)
Not particularly. The one thing that helps but most people don't know about reddit is the fact that adding a .json to the end of each url displays the content of that page in json format.

for example: reddit.com/r/funny.json

This make crawling/fetching content from reddit much more trivial than old school web crawling.

Re:Show HN: I built an auto aggregating bot, collecting tren... by zackKidd on Thursday September 15, 2016 @10:08PM (#12511437)
Is Reddit the only source? If so, do you plan to add any more?
Re:Show HN: I built an auto aggregating bot, collecting tren... by rezashirazian on Monday September 19, 2016 @04:13PM (#12534314)
I have written a crawler for buzzfeed and 9gag but I don't have them running because there is a lot of overlap between 9gag and reddit and they require more work.

To be honest this was a weekend project I did awhile ago. If there is viable interest I may pursue expanding it. But for now I'm going to keep it as is.