News for hackers. Stuff that matters.
Re:Introducing the Priceonomics Business Model: Data Crawlin... by rgovind on Tuesday November 26, 2013 @11:31PM (#6806406)
I believe you have to look at robots.txt file. If it lets you crawl, you should be fine.
Re:Introducing the Priceonomics Business Model: Data Crawlin... by qq66 on Wednesday November 27, 2013 @03:48PM (#6811040)
Does the robots.txt have any legal significance? I would think that the Terms of Use can be more thorough and are more legally relevant.