How to use Chrome to display a website as a Googlebot

Neural coupling and the effect in web design

Table of contents

Introduction to Googlebot spoofing

In this article I describe how and why to use Google Chrome (or Chrome Canary) to display a website as a Googlebot.

We will set up a web browser specifically for Googlebot browsing. Using a user agent browser extension is often sufficient for SEO audits, but additional steps are required to get as close as possible to emulating Googlebot.

Why should I look at a website as a Googlebot?

For many years, we technical SEOs had it easy when testing websites, because HTML and CSS were the main languages of web design. JavaScript was generally used for embellishments (like small animations on a web page).

Increasingly, however, entire websites are being created with JavaScript.

Originally, web servers sent complete websites (fully rendered HTML) to web browsers. Today, many websites are rendered client-side (in the web browser itself) - be it Chrome, Safari or the browser used by a search bot.

From an SEO point of view, some search robots cannot render JavaScript, so they do not see web pages created with it. Especially compared to HTML and CSS, rendering JavaScript is very expensive. It takes up a lot more of a device's processing power - which drains the device's battery - and a lot more of Google's, Bing's or another search engine's server resources.

Even Googlebot has trouble rendering JavaScript and delays rendering JavaScript beyond the initial URL discovery - sometimes for days or weeks, depending on the website. When I see "Discovered - currently not indexed" in the Coverage (or Pages) section of Google Search Console for multiple URLs, in most cases the site is JavaScript-rendered.

To get around potential SEO problems, some websites use dynamic rendering so that each page has two versions:

  • A server-side representation for bots (like Googlebot and bingbot).
  • A client-side rendering for the users of the website.

In general, I think this type of website is too complicated and causes more technical SEO problems than a server-side rendered or traditional HTML website. A small rebuke at this point: there are exceptions, but in general I think client-side rendered websites are a bad idea. Websites should be designed to work on the lowest common denominator of a device, using progressive enhancements (through JavaScript) to improve the experience for people using devices that can handle extras. I will explore this further, but my experience is that client-side rendered websites are generally more difficult to use for people who rely on accessible devices such as screen readers. There are cases where technical SEO and user-friendliness overlap.

Technical SEO is about making websites as easy as possible for search engines to crawl, render and index (for the most relevant keywords and topics). Like it or not, the future of technical SEO, at least for now, involves a lot of JavaScript and different website renderings for bots and users.

When we look at a website as a Googlebot, we can see discrepancies between what a human sees and what a search bot sees. What Googlebot sees does not have to be identical to what a person sees in a browser, but the main navigation and the content you want the page to rank for should be the same.

This is exactly where this article comes in. For a proper technical SEO audit, we need to see what the most common search engine sees. At least in most English-speaking countries, that is Google.

Can we see exactly what Googlebot sees?

No.

Googlebot itself uses a (headless) version of the Chrome browser to display web pages. Even with the settings suggested in this article, we can never know exactly what Googlebot sees. For example, there are no settings for how Googlebot processes JavaScript web pages. Sometimes JavaScript is interrupted, so Googlebot may see something other than what was intended.

The goal is to emulate Googlebot's mobile-first indexing as closely as possible.

In the review, I use my Googlebot browser along with Screaming Frog SEO Spider's Googlebot spoofing and rendering and Google's own tools such as URL inspection in Search Console (which can be automated with SEO Spider) as well as the render screenshot and code from the Mobile Friendly Test.

Even Google's publicly available tools don't show 100 % exactly what Googlebot sees. But together with the Googlebot browser and SEO Spider, they can point out problems and help with troubleshooting.

Why use a separate browser to view websites as Googlebot?

1. convenience

Using my own browser saves time. Without having to rely on or wait for other tools, I can get an idea of how Googlebot sees a website within seconds.

When auditing a website that provided different content for the browser and Googlebot, including problems with inconsistent server responses, I had to switch between the default browser user agent and Googlebot more often than usual. However, constantly switching the user agent with a Chrome browser extension was inefficient.

Some Googlebot-specific Chrome settings are not saved or transferred between browser tabs or sessions. Some settings affect all open browser tabs. For example, disabling JavaScript may cause websites in background tabs that rely on JavaScript to stop working (e.g. task management, social media or email apps).

Apart from a programmer who can program a headless Chrome solution, setting up the "Googlebot browser" is an easy way to fool Googlebot.

2. improved accuracy

Browser extensions can affect the appearance and performance of websites. With this approach, the number of extensions in the Googlebot browser is reduced to a minimum.

3. forgetfulness

It's easy to forget to turn off Googlebot spoofing between browsing sessions, which can cause websites not to work as expected. I have even been blocked from websites for Googlebot spoofing and had to email my IP address to get it unblocked.

For which SEO audits is a Googlebot browser useful?

The most common use case for SEO audits are probably websites that use client-side rendering or dynamic rendering. You can easily compare what Googlebot sees and what a normal website visitor sees.

Even with websites that don't use dynamic rendering, you never know what you'll find when you mimic Googlebot. After more than eight years of testing e-commerce websites, I'm still surprised by problems I didn't encounter before.

Sample Googlebot comparisons for technical SEO and content audits:

    Is the main navigation different?

    Does Googlebot see the content you want indexed?

    If a website is rendered with JavaScript, is new content indexed immediately or so late that its impact is compromised (e.g. upcoming events or new product offerings)?

    Do URLs return different server responses? For example, incorrect URLs can return 200 OK for Googlebot, but 404 Not Found for general website visitors.

    Is the page layout different from what the general website visitor sees? For example, I often see links as blue text on a black background when I fake the Googlebot. While machines can read such text, we want to present something that looks user-friendly to Googlebot. If it can't render your client-side website, how will it know? (Note: a website can be displayed as expected in Google's cache, but that is not the same as what Googlebot sees).

    Are websites redirected according to location? Googlebot mostly crawls from US IPs.

It depends on how in-depth you want to go, but Chrome itself has many useful features for technical SEO audits. I sometimes compare the console and network tab data for a general visitor with that of a Googlebot visit (e.g. Googlebot might be kept away from files that are important for page layout or needed to display certain content).

How to set up your Googlebot browser

Once set up (which takes about half an hour), the Googlebot browser solution makes it easy to quickly view web pages as Googlebot.

Step 1: Download and install Chrome or Canary

If Chrome is not your default browser, use it as your Googlebot browser.

If Chrome is your default browser, download and install Chrome Canary. Canary is a development version of Chrome in which Google is testing new features. It can be installed and run independently of the standard version of Chrome.

Named after the yellow canaries used to detect toxic gases in mines, Canary is easily recognisable with its yellow icon in the Windows taskbar:

Since Canary is a development version of Chrome, Google warns that Canary "may be unstable". However, I have not yet had any problems when using it as a Googlebot browser.

Step 2: Install browser extensions

I have five browser extensions and a bookmarklet installed on my Googlebot browser. I will list the extensions and then go into the settings and explain why I use them.

For Googlebot emulation (the links are the same whether you use Chrome or Canary):

    User-Agent Switcher

    Web developer

    Windscribe (or a VPN of your choice to simulate the Googlebot location)

Not required to emulate Googlebot, but my other favourites for technical SEO review of JavaScript websites:

    Link Redirect Trace

    View of the rendered source text

    NoJS Side-by-Side Bookmarklet

User-Agent Switcher Extension

User-Agent Switcher does what it says on the note: it changes the user agent of the browser. Chrome and Canary have a User-Agent setting, but it only applies to the tab you are currently using and is reset when you close the browser.

I take the Googlebot user agent string from Chrome's browser settings, which is the latest version of Chrome at the time of writing (note that I take Chrome's user agent below, not Canary's).

To get the user agent, access Chrome DevTools (press F12 or use the hamburger menu at the top right of the browser window, then navigate to More Tools > Developer Tools).

    Go to the Network tab

    Via the hamburger menu Network top right: More Tools > Network Conditions

    Click on the "Network Conditions" tab that appears further down in the window

    Deactivate the check box "Use browser default".

    Select "Googlebot Smartphone" from the list and copy the user agent from the field below the list to the User-Agent Switcher extension list .

 Don't forget to switch Chrome back to its default user agent if this is your main browser.

        If you are using Chrome (and not Canary) as your Googlebot browser, you can also tick "disable cache" at this stage (more on this later).

To access the User-Agent Switcher list, right-click on the icon in the browser toolbar and then click Options (see screenshot below). "Indicator Flag" is a text that appears in the browser toolbar indicating which user agent has been selected - I chose GS to mean "Googlebot Smartphone:".

I have also added Googlebot Desktop and the Bingbots to my list.

Why fake the Googlebot user agent?

Web servers recognise who is browsing a website by the user agent string. For example, the user agent for a Windows 10 device using the Chrome browser is as follows:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36

Web Developer Extension

Web Developer is an indispensable browser extension for technical SEOs. In my Googlebot browser, I switch between disabling and enabling JavaScript to see what Googlebot might see with and without JavaScript.

Why disable JavaScript?

Short answer: Googlebot does not/will not execute JavaScript when it crawls a URL for the first time. We want to see a web page before JavaScript is executed.

Long answer: That would be a completely different article.

Windscribe (or another VPN)

Windscribe (or a VPN of your choice) is used to spoof Googlebot's location in the US. I use a pro Windscribe account, but the free account allows up to 2GB of data transfer per month and includes US locations.

I don't think the specific US location matters, but I'm pretending Gotham is a real place (in a time when Batman and co. have eliminated all the bad guys):

The Windscribe browser extension displays the location New York: Gotham, with a background of the flag of the United States of America behind a blue overlay.

Make sure that settings that may affect the display of web pages are disabled - the Windscribe extension blocks advertising by default. The two icons at the top right should show a zero.

For the Googlebot browser scenario, I prefer a VPN browser extension to an app because the extension is specific to my Googlebot browser.

Why fake the location of Googlebot?

Googlebot crawls websites mostly from US IPs, and there are many reasons to fake Googlebot's primary location.

Some websites block or display different content depending on their geographical location. For example, if a website blocks US IPs, Googlebot may never see the website and therefore cannot index it.

Another example: Some websites redirect to other websites or URLs depending on their location. If a company has a website for customers in Asia and a website for customers in America and redirects all US IPs to the US website, Googlebot would never see the Asian version of the website.

More useful Chrome extensions for checking JavaScript websites

With Link Redirect Trace, I can see at a glance what server response a URL returns.

The View Rendered Source extension allows for easy comparison of raw HTML (what the web server delivers to the browser) and rendered HTML (the code that is rendered in the browser on the client side).

I also added the NoJS side-by-side bookmarklet to my Googlebot browser. It compares a web page with and without JavaScript enabled in the same browser window.

Step 3: Configure the browser settings to emulate Googlebot

Next, we will configure the Googlebot browser settings to match what Googlebot does not support when crawling a website.

What does Googlebot crawling not support?

    Service Worker (since people who click on a page from the search results may never have visited it before, it makes no sense to cache data for later visits).

    Permission requests (e.g. push notifications, webcam, geolocation). If the content relies on any of these requests, Googlebot will not see that content.

    Googlebot is stateless and therefore does not support cookies, session storage, local storage or IndexedDB. Data can be stored in these mechanisms, but is deleted before Googlebot crawls the next URL on a website.

Step 3a: DevTools settings

To open the Developer Tools in Chrome or Canary, press F12, or navigate to More Tools > Developer Tools from the hamburger menu in the top right:

The Developer Tools window is usually docked in the browser window, but sometimes I prefer it in a separate window. To do this, change the "Dock Page" in the second hamburger menu.

Deactivate cache

If you use regular Chrome as your Googlebot browser, you may have already done so.

Otherwise, via the DevTools hamburger menu, click on More Tools > Network Conditions and activate the "Disable Cache" option:

DevTools screenshot with the actions described above for deactivating the cache

Block Service Worker

To block service workers, go to the Application tab > Service Worker > activate the option "Bypass for Network":

Step 3b: General browser settings

In your Googlebot browser, navigate to Settings > Privacy and Security > Cookies (or visit chrome://settings/cookies directly) and select the "Block all cookies (not recommended)" option (isn't it fun to do something "not recommended"?).

Also, in the Privacy and Security section, select Website Settings (or visit chrome://settings/content) and individually block Location, Camera, Microphone, Notifications and Background Sync (and probably anything else that appears there in future versions of Chrome).

Step 4: Emulation of a mobile device

Since our goal is to emulate Googlebot's mobile-first crawling, emulate a mobile device in your Googlebot browser.

In the top left of DevTools, click the device toolbar button and then select a device to emulate in the browser (you can also add other devices):

Screenshot of the emulation of a mobile device in Chrome

Regardless of which device you choose, Googlebot does not scroll on web pages and instead renders in a window with a large vertical height.

I recommend testing websites in desktop view and on real mobile devices too, if you have access to them.

How about thinking of a website as a bingbot?

To create a Bingbot browser, use a recent version of Microsoft Edge with the Bingbot user agent.

Bingbot is similar to Googlebot in terms of what it does and does not support.

Yahoo! Search, DuckDuckGo, Ecosia and other search engines are either powered by or based on Bing Search, so Bing is responsible for a higher percentage of searches than many people realise.

Summary and concluding remarks

So, now you have your own Googlebot emulator.

Using an existing browser to emulate Googlebot is the easiest way to quickly view web pages as Googlebot. It is also free, provided you are already using a desktop device that can install Chrome and/or Canary.

There are also other tools that help to "see" what Google sees. I like to test Google's Vision API (for images) and the Natural Language API.

Reviewing JavaScript websites - especially if they are dynamically rendered - can be complex, and a Googlebot browser is one way to simplify the process.

Share this post
Share on facebook
Share on linkedin
Share on twitter
Share on email
More articles
Seo tips for beginners
SEO
8 SEO tips that everyone should know

In 2020, businesses in the United States spent $79.27 billion on SEO services. While SEO services can help your brand drive traffic to your website, the

Blog
The effect of storytelling in web design

Storytelling is deeply rooted in every civilisation. Ancestors have always told stories to younger generations to tell the story of their people.

en_GBEnglish