Re:Show HN: Sitebulb, a website crawler and auditor for SEOs
by hathawayp on Thursday September 28, 2017 @04:07PM
(#15360285)
It should stick the subdomain you specify in the start URL, unless there is a redirect or something. Other subdomains won't be crawled, although it will HTTP status check links to subdomains. So possibly that is what you saw in the URL log on the crawl progress page?
If you want me to take a closer look send the subdomain over to [email protected] and I'll see what's going on.
If you want me to take a closer look send the subdomain over to [email protected] and I'll see what's going on.