Download Google Search Queries Using Google Search Console API

“Farewell, CSV downloads” Google states on Webmaster Central Blog. Login method has changed from ClientLogin to OAuth, your download tool has stopped working, CSV export button disappeared from web interface. So how can you adapt to these changes?

Google Search Console API

Google introduced Search Analytics API. Using new API you can get all data available earlier and much more! For instance, now you can get 5000 top queries per day instead of 1000 earlier. And now you have all the power of segmentation and filtering.

Google provides few libraries for Google Search Console access. But not all of them are really good. PHP library is still beta and there is lack of some features. But Python library is OK.

Let’s use this library and download all search queries for all properties of our Google Search Console account. We’ll download daily data in CSV format.

Prerequisites

Install Python 2.7. Python available for Windows, Mac and Linux. https://www.python.org/downloads/
Install api client. On windows press Win+R and type: pip install --upgrade google-api-python-client
Create or select a project in the Google Developers Console and enable the API .
Download your credentials file.

Choose Credentials on left sidebar.

download cridentials file for your application

Click Download button on the right.

Rename downloaded file to client_secrets.json
Download file to the same folder as client_secrets.json

First Run

Run your script python download.py 2015-09-20 where date format is YYYY-MM-DD.

New browser window will be opened. Click Allow button.

Our script starts to work. All results will be saved to Output folder in CSV format. Notice, that you have webmasters.dat file in your folder. Now you don’t need to allow permissions in your browser.

Under the Hood

The main part is composing request to service with web property parameter.

        # Get top queries for the date range, sorted by click count, descending.
        request = {
            'startDate': date,
            'endDate': date,
            'dimensions': ['query'],
            'searchType': search_type,
            'rowLimit': 5000
        }
        response = execute_request(service, web_property, request)

To get results for a given day we set startDate and endDate to the same value. If we are interested in queries, we set dimensions to query. Other options are country, device, page.

We set searchType to get separate data for all types of searches: web, image, video. We are limiting results with rowLimit . A good point about it is that you get as many as 5000 results! That’s cool!

We are sending such a request for each day, each search type and each web property. List of web properties is available through API:

  # get all verified properties
  siteEntry = service.sites().list().execute()
  properties = [s['siteUrl'] for s in siteEntry['siteEntry'] if s['permissionLevel'] != 'siteUnverifiedUser']

Of course, you can customize script for your needs and automate your repetitive tasks. Please refer to documentation for search analytics and test your requests with API Explorer.

If you ever need custom development with Google API integration please Send us a Request.

SEND REQUEST

Bottom line

Anyway, you have to migrate and get used to new API. Despite any difficulties, you’ll get a lot of new opportunities.