Yeah there's definitely more that can be done in this regard. Right now I have fairly limited statistics on the rate of change of the domains. There's probably improvements to be made by making more informed choices based on that sort of data. Could likely adjust the crawling behavior based on historical change data.