What do you like to use for your scraping? I used python/BeautifulSoup back in the day but there may be better options now. Did you spoof other browser headers or did people really not care?