News for hackers. Stuff that matters.
Re:Ask HN: Do you have a side project you want to sell? by samaybhavsar on Monday November 07, 2016 @02:02PM (#12893836)
http://netstati.com - website profiler
Re:Ask HN: Do you have a side project you want to sell? by wusatiuk on Monday November 07, 2016 @02:49PM (#12894339)
what is your current stack, hosting and crawling infrastructure? how many websites have you crawled so far?
Re:Ask HN: Do you have a side project you want to sell? by samaybhavsar on Monday November 07, 2016 @10:39PM (#12897412)
PHP and Python. Current count - 902873

I have also scrapped all the websites from Quantcast but haven't scanned it yet.

The website is build using PHP including the crawler which works for the new websites not in the list.

For scanning multiple websites, the crawler is in Python which uses multiprocessing.