Yeah, that's good idea - I need to add that to my suggestions for how to implement this.

It should definitely be optional. I can only imagine some busybody PM insisting they block harmless scrapes.

If you're scraping any significant amount of data (>500K), and depending on the frequency, you might also want to add etag/cache-control headers as well as accept-encoding, to save server bandwidth.

Collecting 1 kB every minute might not be a big deal, but collecting 1 MB every minute would cost an AWS-hosted service >$40/year in additional data transfer costs

It's tough when there's a cat and mouse game to spoof your UA so you don't get blocked. I wish webmasters had better relationships with scrapers and could accept the realities that your data will be scraped no matter how much you try and stop it.

IMO, We should really just get rid of the user agent header altogether.