News for hackers. Stuff that matters.
Re:Nordstrom's robots.txt by Springtime on Friday February 27, 2015 @04:58PM (#9121628)
Disallow is used to exclude certain URLs from being crawled, in this case it looks like a search query as the store's search uses this pattern: http://shop.nordstrom.com/sr/query .

There are no comments on this comment yet.