Re:Nordstrom's robots.txt
by Springtime on Friday February 27, 2015 @04:58PM
(#9121628)
Disallow is used to exclude certain URLs from being crawled, in this case it looks like a search query as the store's search uses this pattern: http://shop.nordstrom.com/sr/query .
[ Parent
| on HN ]
There are no comments on this comment yet.