I made a little script that scrapes Google results so I could check my site's keyword positions in Google. It works great, but I only recently found out that what I'm doing is against Google's TOS.

However, there are tools out there (such as the DigitalPoint tracker tool) that have tons of users and must not be violating Google's TOS since Google could easily identify and shut them down. I'm wondering, how I can scrape Google search results without violating the TOS?

I found the JSON/Atom Custom Search API which appears to be a paid service for over 100 queries/day. Would using this API, or another Google API, allow me to use my tool without violating Google's TOS?

Yep that's why Google created the APIs

If you used the DP tool you would have had to register your OWN API key which allows YOU, as the logged in user to perform up to a thousand queries via the API per twenty four hours.

Don't forget though, that the API queries do NOT come from the same datacentres as the web search or toolbar queries come from.

Thanks, Chris!

I was reading the API TOS today and noticed the following:

2.2 Interaction with CS Servers. The CS Implementation must not interfere with the normal functioning of the CS Servers, and Customer acknowledges that Google may monitor CS API activity in order to prevent such interference. Except as expressly permitted by the API Documentation, Customer will not use any automated means (such as the use of scraping, robots or load testing) to access, query or otherwise collect CS related information. Google may set usage limits on features of the CS API.

Does that mean I would be violating the TOS (and the other trackers would be too), since such a tool would be considered "automated"?

The API documentation tells you what is allowed and what isn't. But basically "automated" means repetitive queries or bulk queries.

The idea of using the API is not for firing off massive amounts of queries, but rather for getting search data in different formats.

If you are going to write your own, you'll definitely want to add some "human emulation" features. Most ranking tools have this. Basically you have one setting that states the minimum number of seconds to wait between search queries (or moving to the next page. Then a second setting for a maximum random delay to wait between search queries (or moving to the next page).

For example, if MIN SEC BETWEEN QUERIES is 15 seconds and MAX RANDOM DELAY is 20 seconds, then your software would wait between 15 and 35 seconds between search queries (or moving between pages).

It's silly though to try to write your own these days. With all of the personalization that goes on at Google these days (like geo-targeting) it's MUCH better to buy a tool which was written by someone who knows what they are doing.

Originally Posted by Social-Media

If you are going to write your own, you'll definitely want to add some "human emulation" features. Most ranking tools have this. Basically you have one setting that states the minimum number of seconds to wait between search queries (or moving to the next page. Then a second setting for a maximum random delay to wait between search queries (or moving to the next page).

For example, if MIN SEC BETWEEN QUERIES is 15 seconds and MAX RANDOM DELAY is 20 seconds, then your software would wait between 15 and 35 seconds between search queries (or moving between pages).

It's silly though to try to write your own these days. With all of the personalization that goes on at Google these days (like geo-targeting) it's MUCH better to buy a tool which was written by someone who knows what they are doing.

I'm not trying to do anything shady, though That's why I'm trying to make sure I don't violate Google's TOS.

Originally Posted by chrishirst

Don't forget though, that the API queries do NOT come from the same datacentres as the web search or toolbar queries come from.

As you said would happen, it seems that the data from the API is different than from a Google.com search. Does this happen because the API datacenters are not updated as often or something?

The whole database and index system is in a permanent state of churn called everflux, where there is no "master" database, and the servers and datacentres are continually receiving data from crawlers, other database and index servers.
The system is never sychronised (see Rankings and the Google "Dance").

The API query servers/datacentres are in the same system but API and toolbar queries are routed away to keep the load balanced.

do nslookup queries on

google.com
search.google.com
toolbarqueries.google.com

and you see they have low TTL and expire periods to allow them to rotate the servers/DCs very quickly (use set q=soa to see time reports)

i have a free api of yahoo for the same type of work. and i use another tools for every search engine. it work like a charm.