I also built a rank tracker (WhooshTraffic at the time - no longer around) and we scraped the Search Result Pages as the user would see them, spoofed the session cookie and used human captcha solvers. We would then cache the cookies generated from the captcha solve.

It was highly effective and very scalable because you can stimulate the captcha pages easily and out-of-band generate a very large cookie pool by captcha solving. Then when you go to actually scrape google you can balance your pool of IPs and cookies, chilling them before they trigger another captcha, to handle spiking demand. Contant, sustained demand was very easy to plan for.

Anyway, we stopped using the API after the third day and realized it was not only not accurate and you couldn't turn many knobs (want to scrape results as they look for a different geographical region?)

This is super interesting to me, is there anything else you can share about how you approached this? In my scraping Google experience I have found roughly the same thing where once you've passed the captcha test, you can scrape a lot more.

Were you scraping with real browsers or something like Mechanize/Curl? Rate limiting at all? Proxies or real servers?

We had a pool of IPs we were leasing on our own, proxy services get abused and are poisoned.

We didn't rate limit, we would just increase the size of the cookie pool if a captcha was hit, which was rare because we would scrape n-pages till a threshold was met to prevent that session from being captcha'ed so we wouldn't have to captcha solve it. We had two pools, the primary pool and the "chilling" pool, cookies near their captcha life would cool off for a few hours before returning to the active pool which behaves just like any other resource pool, every page scraped would "borrow" a cookie out of the pool, customize the encrypted location key, and make the request with a common user agent string.

Scaling it was difficult but once we had it figured out, Erlang was invaluable to us and our dependence on IPs dropped once we figured out the cookie methodology.

Solving captchas is cheaper than renting IPs.

Thank you for the reply! Sorry but one last question: can you share anymore about this statement "customize the encrypted location key"?

When you set the location in Google it customizes the cookie with a named field that is a capital L I believe. That field is encoded or encrypted and I could never figure it out so I just constructed a rainbow table by using phantomjs to set a location and scrape the cookie out, pairing the known location value with the encrypted value so that we could customize "the location of the search".

Oh and I just used a generic HTTP request client in Erlang and xpath / HTML parser to extract what we needed.

God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.

It was indeed pretty rough it wouldn't surprise me if Google moves to js generated dom elements to combat rank trackers, at the time it was fine because they want to service non-js browsers but that might change.

Parsing it wasn't hard but it wasn't fun...

Wouldn't it be easier if they generated the dom elements via JS? That would imply that they're getting a JSON or something like, parsing it and creating the DOM.

No because then you'd have to use a headless browser that can execute js. That increases time and cost when scraping, though it wouldn't surprise me if it ends up going that way.

Why execute the JS? Just parse the JSON (I'm just guessing here). No need to mess with DOM or JS.

From my last and admittedly cursory look it wasn't that simple. IIRC JSON wasn't even used, I think it was a binary protocol.

If you have an HTML parser, it's just a matter of selecting the right DOM nodes.

Def not that easy with Google. It is not uncommon to see 20+ serp variations on a given day if you are crawling at high volume, changing user agents, etc. The whole process is fairly terrible to parse consistently.

Yes, but if your markup is spaghetti, "selecting the right DOM nodes" becomes a lot more difficult.

Having built a ranktracker before, and still being good friends with people who run one of the biggest rank trackers on the market, using the google API's simply isn't an option.

Simply put, the API never returned the results in the exact same order as the actual search results did.

The best / most reliable rank trackers did (and still do) simply use proxies to get around the google captchas. I've scraped millions of pages from google over the years, and with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily.

This is especially true with the dropping cost of proxies. I'm obviously an exception considering I run several scraping SaaS and have generally specialized in it for years, but I can he hitting x.com with 30,000 different IP's within the hour.

Any additional tricks you want to share and/or cheap proxy services you'd recommend? Do you make use of selenium and/or phantomjs?

Haven't all the web search APIs been gimped? In a past life this interested me greatly and I turned to both Google's API and to Yahoo's BOSS and they both basically sucked. You could tell you were not getting the type of results a proper search would return.

> with enough proxies, the correct proxy delays/timeouts, and other little tricks you pick up along the way; you can actually scrape google pretty easily

Try putting a site: in the search and it becomes suspicious fast.

You're correct. However for Rank Tracking these types of queries aren't needed.

That is probably why some rank trackers give different results. Some probably use the API and others scrape Google's result page.

I'm not acquainted with RankScanner, but my educated guess is that they use one of Google's (many) authenticated APIs[1] for search/content categorization/ranking analysis.

[1]: https://developers.google.com/custom-search/

Thanks. I was under the search impression that this was only for building your own internal search. do you know how much the pricing is?

There are some prices listed here: https://developers.google.com/custom-search/json-api/v1/over...

These seem to be for products in a slightly different domain than plain old search, though.