> a good workaround is setting your user agent to the Googlebot

Be sure to follow the site's robots.txt rules, though. Otherwise, you could end up getting googlebot banned! /s