Increasing Your Google Crawl Rate

May 4, 2007 – 6:54 pm

by Darren

Okay, today we’ll talk about increasing your Google Crawl Rate. Some of you may be wondering, what the hell is such a thing? You’re saying “Huh, I never heard of it. Can it really be important to me?” I have to answer “Yes!”. I wouldn’t waste your time writing up some crap just to say I “posted” today. So let’s get to work.

What is the Google Crawl Rate, and why should I care?

Good thing you asked! Google’s crawl rate keeps a historical record of how many pages they’ve crawled on your website in the last 90 days. They show you the minimum, average, and maximum number of pages they “fetch” from your website on any given day. As a blogger, you want that number to be heading higher, especially as you look at the graph over time. The more pages you create, the more incoming links you attract, and the higher your crawl rate goes.

Google doesn’t mention anything particularily sinister in its’ description of the Crawl Rate, but the basic idea is, Google uses the Crawl Rate to determine how worthy the pages on a website are and then they set a crawling schedule appropriately. If they can tell your website infrequently updates, they’ll assign a higher cost to crawling your pages and do it less frequently. If you have a high crawl rate, it means you have lots of fresh content that will be added to Google’s index fast.

How do I increase the Crawl Rate for my blog?

Here’s the Crawl Rate for Absolutely Fabulous - The Celebrity Blog

As you can see, the Crawl Rate has increase rapidly recently. The primary reason: the quite obvious fact of many more updates done more frequently. The effects this has had are:

  • Crawl Rate has increased and pages are added nearly instantly to Google’s index
  • Revenue has increased
  • Visitor participation has increased

The logic is straightforward, so I won’t belabor the point: “The More You Shoot, The More You Score. The constant push forward is making the numbers go up, as if through sheer force of will. Archive size increases, and with it, first time visitors and revenue increase. It seems like a simple enough formula.

Do you pay attention to your Crawl Rate?

  1. 2 Responses to “Increasing Your Google Crawl Rate”

  2. Googlebot,

    I tried your method and it told me that it was an invalid option. Thanks though.

    Are there any other ways to get Google to crawl faster?

    Tim

    By Tim on May 30, 2007

  3. faster isn’t better (that’s what she said)

    i think if you have a way to make minor changes to all your pages frequently, you will see the crawl rate increase, but the ranking will remain the same. it’s not because your site is important, it’s because the content is changing constantly.

    By Maya on Jun 18, 2007

Sorry, comments for this entry are closed at this time.