Show HN (again): Web-based Link Checker with Blacklist Lookup

Re:Show HN (again): Web-based Link Checker with Blacklist Lo... by symkat on Wednesday December 28, 2011 @12:24PM (#3400496)

Looking at my logs I only see 5 requests (maybe I'm grepping for the wrong thing?) but the UI shows many more links checked.

  grep www.drlinkcheck.com  /var/log/nginx/access.log | wc -l
  5

All of those have the user agent 'Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/)

Also, I think it would be useful to add a section in 'About' that says the user agent the crawler will use.

[ Parent | on HN ]

Re:Show HN (again): Web-based Link Checker with Blacklist Lo... by awulf on Wednesday December 28, 2011 @12:56PM (#3400627)

That's strange, your grep looks fine to me. Is it possible that you have configured nginx to use different log files for different locations (or maybe don't log at all)? Other than that, I don't have any good explanation.

[ Parent | on HN ]

Re:Show HN (again): Web-based Link Checker with Blacklist Lo... by symkat on Wednesday December 28, 2011 @01:26PM (#3400728)

Oops, I'm a dumby and hadn't had enough coffee yet. That file was just logging 301s, and the actual target was going elsewhere, which has an appropriate number of requests.

Overall pretty awesome, thank you!

[ Parent | on HN ]