Frustrated by the (not provided) search queries in your Google Analytics organic traffic reports? There are ways by which you can still make sense of that data. This post explains how, using Google Analytics and Search Console.
Do you know Google Analytics used to provide the exact organic search keyword in the good old times? Frustrating as it is, GA no longer provides you the organic search terms. The decision on this came on 18 October 2011, when Google switched to secure search.
That move meant the search queries made by users who are at the time logged into their Google accounts were kept secure and not revealed to the Analytics account of the target website.
However, while the search became more secure, Google Analytics users got more frustrated seeing data “(not provided)” and “(not set)” on their reports.
As you can see above, over 98% of the search queries are not provided. There is a lot of intelligence there that you are missing.
In this post, let’s try and understand how you can make sense of that (not provided) data. Now, I’m not promising that I will magically reveal the keywords Google is not revealing, but there are other means to gather better intelligence.
Article Outline
1. Location of Organic Search Keyword Data on Google Analytics
2. Linking & Verifying on Search Console
3. Using Secondary Dimensions to Understand (Not Provided)
4. Creating Filters to Understand (Not Provided)
5. Comparing GA & Search Console Reports
1. Where Can I See the Organic Search Query Data on Google Analytics?
The organic search query data is given as part of the Acquisition data in Google Analytics. Go to Acquisition -> Campaigns -> Organic Keywords. This is where you have the organic search keywords.
Here, the campaigns dimension doesn’t mean Google Ads campaigns. It stands for all kinds of campaigns, including the organic search campaigns.
2. Link and Verify Your Website on Google Search Console
As the first step of setting up your website and tracking it using Analytics tools, verify it on Google Search Console, if you haven’t already done so.
Google Search Console is the tool that helps webmasters manage the search listing of their web properties. It lets you analyze a large number of things related to your website:
- Search queries, clicks, impressions, and CTR
- Links to your website
- Internal link structure
- Mobile usability
- Index statistics
- Errors on your site you may need to fix
- Sitemaps & URL submissions
- Security issues
- Structured data and rich snippets
- Improvements on HTML
You can follow this guide to add and verify your site on Google Search Console.
Once you are verified on Google Search Console, you will have access to the search queries that trigger your site and the number of clicks that they generate. This report can be accessed from Search Traffic -> Search Analytics.
Here are the various details of this report.
Clicks: These are the clicks that send users to your website from a Google search. These clicks usually result in a hit on your website. It could be a new user, existing user, new session, existing session, etc.
Impressions: If one of your web pages shows up on Google search for any keyword, it is counted as an impression. So, by default, the number of impressions has to be equal to or greater than the number of clicks. And for every click, there is at least one impression.
CTR: CTR stands for click-through rate. This is the rate at which users click your impressions. For instance, if your web page shows up ten times for a search query and three times those impressions resulted in a click, your CTR is 3/10 or 30%.
Position: This is the position of your web page on the search result page of Google. If your page appears on the first position, naturally its CTR will be very high. The lower the position, the worse the CTR gets.
3. Use Secondary Dimensions to Make Sense of the (Not Provided) Data
In every GA report, you have the option at the top that says “Secondary Dimension” using which you can further narrow down the report data. It is very useful to split your data with an additional dimension.
Open up your organic search queries report and click the “Secondary Dimension” dropdown at the top. Type “page title” in the search box and select the corresponding dimension.
Once you select, the secondary dimension will be added to the report and it will give you an additional column showing the pages corresponding to the (not provided) visits.
Alternatively, you can use the “Landing Page” dimension as the secondary dimension to identify the landing pages associated with an organic search. This will give you a clear idea of which of your pages actually rank well on Google.
4. Create a Filter on Google Analytics to Pick Apart the (Not Provided) Data
As given in image 7.1, the not provided data is an accumulation of all the secure search queries that trigger traffic to your web pages. So, it is associated with hundreds or thousands of search queries and hundreds of web pages on your site.
If we are able to split it in the first level, we will get a little bit more intel from it. In this step, we will see which web pages were triggered by the not provided queries. For this, we need to create a Google Analytics filter.
What we will see is how we can replace “(not provided)” with the actual title or the URL of the page itself, rather than adding a secondary dimension to the report. So, we are going to modify the primary dimension itself, only for not provided queries, not for all queries.
For instance, if a not provided search query results in a page view to “internet of things services” page, we will see the page title in place of “(not provided)”. This way, we will be able to split the 98% not provided data by a number of rows with page title in place of (not provided).
In order to create the filter, click the Admin icon (gear) on your Google Analytics dashboard and click “All Filters” under the Account section. Here, click the “Add Filter” button.
Give a detailed name for your filter. Configure the filter in the following way.
- Filter Type: Custom->Advanced
- Field A->Extract A: Campaign Term: (not set)|(not provided)
- Field B->Extract B: Page Title: (.*)
- Output To->Constructor: Campaign Term: NP: $B1
The final output to the campaign term is “NP: $B1”. $B1 indicates the extracted page title using a regular expression (field B). The prefix “NP” is for us to understand that it was a not provided search term.
If you want to use something other than page title as the replacement string, you can select it by changing the Field B.
You can see the results in the image 7.7 below; in my case the page URL replaces “(not provided)”.
5. Comparing the Data Between Google Analytics & Search Console
Now, in order to get an idea of which might be those actual keywords, you want to compare the GA data and the Search Console data.
After a few days of activating the filter, you will have enough data to compare. Pull the search keyword report from the Campaign -> Organic Keywords section.
At this time, the original 5,217 not provided data have been split into hundreds of rows with the page URL or page title as the value for the keyword.
Now, we can see that the home page accounts for over 3,300 users among the 5,200+ not provided data. We still don’t really know which keywords (search queries) referred those users.
In order to find the search queries, you can use Search Console.
Go to Search Console and open up the Search Analytics report. Set the date range to the same as in Google Analytics, by selecting the custom date range option.
But the data that we see on Search Console and Google Analytics can be quite disparate. While we see 5,217 users and 6,717 sessions on GA, we see 6,304 clicks on Search Console.
This disparity has a lot to do with several aspects:
- Your GA view might have other filters set up to exclude traffic from your ISP.
- GA users could cause multiple clicks from different searches, but the user data is counted only once in GA.
- A default GA session times out in 30 minutes, but that could be associated with multiple page views through multiple clicks in Search Console.
- Clicks to non-HTML pages (PDF files) on your server are still counted by Search Console, but not by GA.
- If JavaScript is not enabled at the user side, that session is not tracked by GA, but click is still counted by Search Console.
- If a user clicks on a search result and lands on a page where GA tracking code is not implemented, that visit is not tracked by Analytics.
- Search Console doesn’t show you all search queries (see the 4th heading further down).
Despite such disparities, you can trust the Search Console data to provide you with more accurate insights on your search ranking. It’s, hence, pretty valuable.
Let’s see how.
First you need to select the “Queries” option and check Clicks, Impressions, and Position.
This will show you the search queries and their click data, as you can see in the image 7.9 above. By the way, I am masking the keywords because they are the brand name of my company.
Here, you have all the queries, but you still don’t have which pages those queries refer to. One thing you can do is click the query to open Google search on a new tab and look for your web page’s search snippet there.
But this can be quite different from what you expect because of personalization in Google search. This means, the search results you see could be wildly different from the search results someone else from a different country sees. It depends on the URLs you frequently visit, your search history, interests, and various other parameters.
So, what you should do is click on the double-arrow button at the end of each keyword row. It will open the individual report for the keyword.
For instance, I’ll click the double-arrow button for the first keyword, which has 1,572 clicks. In the page that opens, just select the “Pages” radio button at the top and the URL corresponding to the search query will be revealed.
In this case, as you can see the URL corresponds to multiple pages on the site and they have all been ranked at various positions. They have all had clicks on them.
As you can see, for the keyword (which is the company name itself), the home page got an average first position in Google, and as a result, it was visited over 1,437 times, at a CTR of 45%. For brand keywords, CTR always tends to be very high.
But you can also see that some other pages have average position of 1 for the same keyword. You can see that Careers, Life at the company, Insights, Internet of Things, and Contact pages have the same number 1 position in Google.
How do multiple pages from the same domain have the same search ranking position for one keyword?
It’s because all of those pages are part of the site links section in the Google search result. As an example, check out Apple’s search result in Google. The highlighted links are all site links from apple.com, and they all share the number 1 position with Apple’s home page.
In our search result, since the search term is the brand name itself, it corresponds to multiple URLs (including site links) on the search result page. On the other hand, a long-tail keyword would usually correspond only to one URL. Take a look at the image below. For the long search query, “RPA in telecom,” there is only one page ranking at 4.9th position with a CTR of 26.92%.
In short, here are the steps to get pages corresponding to each search query through Google Search Console.
- Get to the Search Analytics report on Search Console.
- For the keyword for which you want to find the target page, click the double arrow button.
- On the report that shows up, select the “Pages” radio button.
6. “Other” Search Queries in Google Analytics Report of Search Console Data
After you connect Search Console to GA, you will have an additional section among the GA reports titled “Search Console.” Here, you have the section “Queries,” where you will see the search query report from Search Console aggregated. Open up this report and compare it with the Search Analytics report in Search Console itself.
See the image below. This is the comparison of the Search Console report available through Google Analytics and the same report on Search Console itself.
You can see that there is a row “other” that accounts for 1801 clicks and over 23,000 impressions. We don’t get this information in Search Console report. This is a piece of missing information that we can’t get. Why is this information filtered out in Search Console report?
The key reasons include:
- They are very low volume search queries
- They could contain private or sensitive information
Official explanation from Google is:
“To protect user privacy, Search Analytics doesn’t show all data. For example, we might not track some queries that are made a very small number of times or those that contain personal or sensitive information.”
Conclusion
It’s difficult to get the search query information now after secure search has become the standard for all kinds of search engines. Bing moved to secure search in 2015; Yandex had done so in 2014 itself. None of the top search engines provide you with the search query data.
But Bing Webmaster Tools and Yandex Webmaster Tools can help you with the search queries still. It’s very important to have your site verified on both those platforms and provide sitemaps to get it indexed.