Fetch as Google: Fix the Common Crawling Problems and Error Messages

In Blogging by Fathi Arfaoui10 Comments

Google webmaster tools added to the “Fetch as Google” option to let site owners manage their blogs and sites easily, and submit their content manually to Google. That’s helpful, especially, if they think that some of their pages are not indexed in the search page.

But sometimes, things can be complicated, especially, for beginners who get different types of error messages, such as “temporarily unreachable” page. Or, in other cases, the page keeps refreshing without any test result and much more.

That’s not a rocket science if you have a total access to your site. You can see the probable things that cause the problem. This post will show you the common problems with Googlebot errors and warnings, that you receive sometimes, and you can’t find a solution to fix them.

What’s Fetch as Google?

Fetching as Google is a way to show webmasters how the Googlebot access their sites, and how it sees their pages exactly as a crawler. So, they can optimize the page on that test, and see if something goes wrong with the test.

In your Google webmaster dashboard, you can access the testing tool, by clicking the “Crawl” menu, then, “Fetch as Google”, as the following screenshot.

fetch as google

The fetch as Google crawl limit is 500 pages per month. So, you can submit all your new posts, and you will never reach your limit in the majority of cases. This is an individual URL submission and only that page can be crawled.

The second option limit is the full home page fetch. In that case, you have a maximum of 10 tests per month. But, Googlebot will follow all the homepage links, and you end with more pages, and that takes more minutes to complete especially, with lots of content.

You can submit your page URL and let Google crawl it instantly, and give you the test result. You can also, use the “Fetch and render” option that will show you the page later when you click on it.

If you see the “pending fetch”, don’t worry about that. Sometimes, Google receives thousands of requests, at the same time, and the bots will not be able to access all these sites at the same time. So, they simply add your URL to the list, in general, crawlers will visit your site in a few minutes.

Temporarily unreachable page

In many cases, bloggers get this error message in their Webmaster tools dashboard. The good news is that you don’t have to worry too much about that unless you host your site with a low quality hosting service.

The message is clear and it means that your server is briefly down, or, the specified URL returns a 501 error, and that’s a server issue that you can’t solve it from your part. Just wait a few minutes and test that page again, and see if you get the successful result.

fetch as google unreachable page

If you’re using a free hosting, you will get that error in the majority of the time, and you know that something for free will never help your business. So, it’s the time to think about a premium hosting service for your WordPress blog or website.

I recommend WP Engine for their high-quality service and fast customer support. No more issues with your server, because the company has technicians that will manage the technical part of your hosting. You just need to create content and never worry about the server and the site speed.

Or, if you need cheaper hosting, but for all the applications and with more flexibility and high quality, then use InMotion Hosting.

If you’re using a security plugin such as Wordfence security or any other one, make sure that Googlebot can access your site and is not blocked. The majority of plugins work fine, but sometimes things can work wrong. You can verify that by disabling all your plugins, and then wait one minute or so and test the URL again.

Continue that process for each plugin, until you find the one that blocks the Googlebot from accessing your site. If you still get the same error after all that, then, contact your hosting company for support.

If your site is new and you’ve submitted it once, make sure that you have your preferred domain name. You can set up that in your account, by following the above link. At the same time, make sure you’ve properly submitted your blog sitemap.

Redirected error

In this case, you have one of these issues, you have entered the full URL, and the Googlebot will fetch the domain. In this case, make sure you just enter the URL after your domain name. So, you just need to enter the post URL, and Google will add your domain name automatically.

Webmaster Tools

The second cause is that you’ve entered the URL without the forward slash “/” at the end. You can read how to fix this redirect problem in Google webmaster tools.

Failed fetch

Failed fetch

This is another common problem with crawling a website, that means no way to access your page. It’s not a redirect and other issues, but a complete problem that needs to be solved. Before doing anything, visit your URL exactly as you typed it in the fetching test, then, see what happens in your browser.

If you see the page correctly, try using the Google page speed insights that will test your page performance, and show you a real thumbnail about that page.

From there, you can see what happens really, and you can even, see the exact error message that appears from that different location.

In most cases, you will find that your server firewall is blocking Google crawlers, and you need to ask your hosting customer support, for help. Explain what happens exactly with a screenshot, at least, they will solve the problem in general.

A tested solution, that fixed the problem, is to reinstall the WordPress. There is no exact explication for that, but, that fixed the issue in many cases. It seems that the Fantastico auto-installer or the other similar tools will add some wrong configurations or permissions. In a few number of installations after the second install, the blog worked properly and Google accesses it naturally.

Page keeps refreshing

That means the web browser is sending wrongs requests in general, or has a spam extension or add-on. If you’re using Google Chrome, reset your browser after cleaning it. Then, scan your full computer for virus and malware.

You can switch to another browser and see what happens. In many cases, the web browser is what causing the refresh problem that gives no results, and you will notice a redirect to the Google home page.

So, reset your browser and remove all your newest untrusted software, and you will fix your problem. If nothing happens with all that, then, use a completely different computer, and test the URL from there.

These are the most popular errors in Google webmaster tools, if you have other messages, you can add them, and it’s better if you add your solution, and how they will be fixed.

Fathi Arfaoui: A Physicist, Blogger, and the founder and owner of Trustiko.com. He shares Business, Blogging, WordPress, Web Safety, and Blogging tips to build better websites and blogs. Also, he shares online marketing strategies and recommendations.

Comments

  1. When I click on a page under renfer & fetch, I see the title: This is how Google saw the page: {image of page}
    Should I be concerned that the page looks like it is behind a cloud? Does that mean Google isn’t seeing the page?

    1. Author

      Hi, Yvette, you need to click the title of the page on the Google fetch option, then, you will be taken to the page as Google see it. That way, you can see the same page that Google get, scroll down and find if there is something missing.

  2. Hello Fathi!

    Wonderful article, you helped us a lot with it, i have a question, when i fetch & render my website, once i click the part to preview my site, the only thing that i see is a blank page, no content at all

    This is how Googlebot saw the page: BLANK

    is this a problem at all?

  3. Hi I have this error while I’m trying to fetch although everything is ok with my site and i’ve been waiting for this for a day.

    My server isn’t down and is premium host but i encounter this problem from last night.
    my robots.txt is ok and is allowed to googlebot.

    please guide me with this problem

  4. I have a website that is showing only the home page as an error when I do a fetch and render. The home page has also been dexindexed. Do you have any insight.

  5. Hi,

    First of all thanks for the great article,.

    Here what my question is if my site showing partial status when I am going to fetch any url. So does it affect on my google ranking or search position ?

  6. Hi,

    I have new wordpress website.

    When I search my web URL in Google at that time the error display “A description for this result is not available because of this site’s robots.txt”

    while robots.txt is not blocking URLs and in webmaster there are same error in sitemap.xml – the URL was blocked by robots.txt file.

    Please note : I already tried fatch as google option but it;s not work at all.

    Please help.

  7. I cannot fetch on three pcs but I can on my iPad. All same settings. However, also robot.txt and settings in Google Search Control fon’t work on the pcs. They do on iPad flawlessly. There is nothingbto be read anout this on the internet. I a. using WordPress but also direct access to the search console triggers the same problem. All other functions in te console work flawless.

Leave a Comment