• Related

  • Deep Crawl and the Google Dance

    The deep crawl is traditionally performed by the main Google spider, called Googlebot. Google updates its main index once a month after the deep crawl is completed for all websites. (Google now tends to go with a more continuous update although monthly updates still happen from time to time).

    This process usually starts the last week of every month and continues for up to a week. The index update is generally based on content they have cached or stored in their database earlier in the month for your site. Because of the nature of the update process, ranking calculations are performed multiple times for each page of every site. Because of the sheer number of web pages in the  Google index, these calculations can take up to a week to complete.

    During this period, search rankings can fluctuate – sometimes minute-by-minute. These monthly fluctuations are termed the Google Dance, as your site’s ranking can jump around on a search results page (or even appear to drop completely). You can view other versions of the index on different servers at the ten major Google datacenters by using the Google Dance Tool.

    Important: It is critical that your website is up and running when Google visits you. If your site is down, your listing on Google may disappear until the next update! The reason is that Google thinks your site no longer exists and may remove it from the index.

    Fresh Crawl

    The fresh crawl is performed by a different Google spider, called Freshbot. Fresh crawls are done several times a week (daily for some sites), but only some pages are crawled.

    Freshbot looks for new pages and for pages whose content has been recently updated. New pages are included in search results right away, which means they can be found by visitors almost immediately, even though they are not yet in Google’s main index. Rather they are stored in a temporary index and then ranked.

    Daily fluctuations in the search results caused by new pages being crawled by Freshbot and then ranked are termed the everflux.

    New pages aren’t added to the main index, and hence don’t have an accurate ranking, until after the main crawl. It is not uncommon for new pages to drop in rank once they are compared against all other web pages in the main index. This is because the pages are then compared against other pages in the main index.

    Note: Over the spring and summer of 2003, Google has been slowly merging their deep crawl and fresh crawl results together in addition to rolling out some other changes. This led to a period of unpredictable or pre-April 2003 rankings for some sites. Although not officially announced by Google, the consensus is that Google is moving toward a continuous update, with the traditional Google Dance and monthly visits going away.

    If interested, you can check your server log files for the user-agent “Googlebot”. This will tell when Google crawls your site. You can also check by IP address although this method is not as accurate as Google uses different IP addresses for their robots, which can change over time.

    Share this to your friends

    2 comments to Deep Crawl and the Google Dance

    • Found the information worthy as I had no idea of posting a comment on the blog,.I keep seeing articles like these.

    • @ Astaga : Uhm, just like we keep seeing so many people sells cellphone credits while you can buy it through the ATM with no additional charge (seller profit).
      I’m just trying to compile the large kind of SEO knowledge. in case, i may learn from my own post and somebody might share their opinion to the good of all.

      by the way, thanks for visiting here

    Leave a Reply




    You can use these HTML tags

    <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

    Maximum 2 links per comment. Do not use BBCode.