Skip to main content

Googlebot Crawl Budget Explained on Google Webmaster Central Blog

Recently, we've heard a number of definitions for "crawl budget", however we don't have a single term that would describe everything that "crawl budget" stands for externally. With this post we'll clarify what we actually have and what it means for Googlebot.

First, we'd like to emphasize that crawl budget, as described below, is not something most publishers have to worry about. If new pages tend to be crawled the same day they're published, crawl budget is not something webmasters need to focus on. Likewise, if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently.

Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters, for example.

Crawl Rate Limit

Googlebot is designed to be a good citizen of the web. Crawling is its main priority, while making sure it doesn't degrade the experience of users visiting the site. We call this the "crawl rate limit," which limits the maximum fetching rate for a given site.

Simply put, this represents the number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches. The crawl rate can go up and down based on a couple of factors:


  • Crawl health: if the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Googlebot crawls less.
  • Limit set in Search Console: website owners can reduce Googlebot's crawling of their site. Note that setting higher limits doesn't automatically increase crawling.


Crawl Demand

Even if the crawl rate limit isn't reached, if there's no demand from indexing, there will be low activity from Googlebot. The two factors that play a significant role in determining crawl demand are:


  • Popularity: URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our index.
  • Staleness: our systems attempt to prevent URLs from becoming stale in the index.


Additionally, site-wide events like site moves may trigger an increase in crawl demand in order to reindex the content under the new URLs. Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.

Factors Affecting Crawl Budget

According to our analysis, having many low-value-add URLs can negatively affect a site's crawling and indexing. We found that the low-value-add URLs fall into these categories, in order of significance:


  1. Faceted navigation and session identifiers
  2. On-site duplicate content
  3. Soft error pages
  4. Hacked pages
  5. Infinite spaces and proxies
  6. Low quality and spam content


Wasting server resources on pages like these will drain crawl activity from pages that do actually have value, which may cause a significant delay in discovering great content on a site.

Top Questions

Crawling is the entry point for sites into Google's search results. Efficient crawling of a website helps with its indexing in Google Search.

Q: Does site speed affect my crawl budget? How about errors?

Amswer: Making a site faster improves the users' experience while also increasing crawl rate. For Googlebot a speedy site is a sign of healthy servers, so it can get more content over the same number of connections. On the flip side, a significant number of 5xx errors or connection timeouts signal the opposite, and crawling slows down. We recommend paying attention to the Crawl Errors report in Search Console and keeping the number of server errors low.

Q: Is crawling a ranking factor?

Answer: An increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it's not a ranking signal.

Q: Do alternate URLs and embedded content count in the crawl budget?

Answer: Generally, any URL that Googlebot crawls will count towards a site's crawl budget. Alternate URLs, like AMP or hreflang, as well as embedded content, such as CSS and JavaScript, may have to be crawled and will consume a site's crawl budget. Similarly, long redirect chains may have a negative effect on crawling.

Q: Can I control Googlebot with the "crawl-delay" directive?

Answer: The non-standard "crawl-delay" robots.txt directive is not processed by Googlebot.

Q: Does the nofollow directive affect crawl budget?

Answer: It depends. Any URL that is crawled affects crawl budget, so even if your page marks a URL as nofollow it can still be crawled if another page on your site, or any page on the web, doesn't label the link as nofollow.

For information on how to optimize crawling of your site, take a look at our blogpost on optimizing crawling from 2009 that is still applicable. If you have questions, ask in the forums!

Comments

Popular posts from this blog

The Philosophy Behind Link Building

Among Internet Marketer, there seems to have controversy when it comes to baclinks, link building strategies & it's importance. if you ask a group of online marketer, What is a backlink?, Why are backlinks So Useful?, How do search engines evaluate backlinks?, importance of backlinks in seo? link earning, etc you'll get different & vague answers from them.

To make you clear, in this post I'll write simple to understand definition of Philosophy Behind Link Building & to help you make sense of what's backlinks truly harmful for your website.

So, What is a Backlink?
A backlink is a clickable piece of text or an image that links one website, webpage, or files to another. As an example here is a link to the Local SEO Consultant at this website, which is surrounded by the anchor text "Local SEO Consultant", So, we’ve got the answer of most important question out of the way.

Backlinks are also knows as Internal Links, Inbound Links & Outbound Links.…

6 Lead Generation Strategies Using White Papers

It’s clear that white papers are a great way to open doors and make the sales process easier. At this point, you’re probably excited about using white papers. Once you have a quality white paper written, it`s time to use it to generate leads. Use these proven lead generation strategies to attract interest from qualified prospects.

Build an email list with your white paper.
Email marketing remains one of the most reliable ways to make sales. There`s just one problem: how do you get relevant prospects to sign up for your email list? There’s a simple solution: offer a free digital copy of your white paper to each person who signs up.

You can then reach out to each person who signs up for the white paper. Pro Tip: Experiment with different services such as the Hello Bar or SumoMe’s Scroll Box to increase the number of email signups on your website.

Introducing a new product to past customers.
How often have you heard the advice – which is absolutely true! – that selling to past customers th…

How To Improve your Online Retail Store’s Search Engine Optimization

You know the math right, According to the National Retail Federation’s annual consumer spending survey conducted by Prosper Insights & Analytics, consumers plan to spend an average of $935.58 during the holiday shopping season. Total spending includes gifts for others, self-spending, food, flowers, decorations and greeting cards for Christmas, Hanukkah and Kwanzaa. Total planned spending this year is second only to the record total spending in 2015, at $952.58.

NRF Expects online retail sales in November and December (excluding autos, gas and restaurants) to increase a solid 3.6 percent to $655.8 billion. Online sales are forecast to increase between 7 and 10 percent over last year to as much as $117 billion. Retailers are expected to hire between 640,000 and 690,000 seasonal workers this holiday season, in line with last year’s 675,300 holiday positions.
In our last article I talked about how November Replacing Black Friday & Cyber Monday for USA, UK & Canadian Shoppers.…