Ask HN: Projects that don't make you money but you're doing it out of sheer joy?

Re:Ask HN: Projects that don't make you money but you're doi... by kaivi on Thursday August 10, 2017 @08:51PM (#14985774)

I'm hoarding profile data diffs from a well known social network. Been crawling every single user for the past 2 years and saving the changes. Had to stop doing it last month, after storage costs became too much.

[ Parent | on HN ]

Re:Ask HN: Projects that don't make you money but you're doi... by j_s on Friday August 11, 2017 @08:40AM (#14989014)

Can you share any of the techniques you use to avoid being blocked for scraping?

Interesting to hear that storage was too expensive. I'm sure you could find personal sponsors here who would help with funding in exchange for access to the data - even with a 'no commercial use' type of restriction!

[ Parent | on HN ]

Re:Ask HN: Projects that don't make you money but you're doi... by SIRHAMY on Thursday August 10, 2017 @08:54PM (#14985787)

Woah, do you have an end goal with this. Willing to let people get their hands on it?

[ Parent | on HN ]

Re:Ask HN: Projects that don't make you money but you're doi... by kaivi on Thursday August 10, 2017 @09:09PM (#14985859)

I'd like to set it up as a subscription service like Domaintools. I'm also crawling and storing user's comments, friends, posts and links to media published in other groups.

It will make a great tool for profiling a person, but I'm not sure on how (il)legal it will be.

[ Parent | on HN ]