Re:Show HN: Sitebulb, a website crawler and auditor for SEOs
by blacksmith_tb on Thursday September 28, 2017 @03:46PM
(#15360136)
Hmm, I set it to crawl our little Ghost blog (e.g. blog.example.com) and it immediately jumped to spidering our main site (e.g. www.example.com). Now, I imagine if I had properly poked at the Advanced settings I could have limited it to the initial subdomain, but I would have expected that to be the default...
[ Parent
| on HN ]
Re:Show HN: Sitebulb, a website crawler and auditor for SEOs
by hathawayp on Thursday September 28, 2017 @04:07PM
(#15360285)
It should stick the subdomain you specify in the start URL, unless there is a redirect or something. Other subdomains won't be crawled, although it will HTTP status check links to subdomains. So possibly that is what you saw in the URL log on the crawl progress page?
If you want me to take a closer look send the subdomain over to [email protected] and I'll see what's going on.