Google posted a new help document on “Things to know about Google’s web crawling.” While many of those “things to know” are already known, Google felt it would be a good idea to make this document in order to provide “basic educational information about crawling to better highlight various resources about crawling that are available to site owners.”
The document has 9 items posted in it right now including:
Frequent crawling is a good sign! Google wrote,
“If we’re crawling your site a lot, it’s an indication your pages have fresh or highly relevant content that people want to find, and that our systems are recognizing that demand. Online shopping is a great example: we crawl ecommerce sites often so that our results will display retailers’ most up-to-date prices, promotions, and inventory status.”
Other items. Here is the full list, but make sure to check out the help document to read it all. None of it is new but it is a helpful refresher:
What is crawling? In short, crawling is how Google “sees” the web
We have many crawlers; they each have important jobs
We perform repeat crawls to find the latest updates and to provide the freshest search results
Frequent crawling is a good sign!
Google’s crawling has grown over time as pages have become more complex
We optimize crawling automatically
Google crawlers never go into paywall or subscription content without permission
Site owners have control over what gets crawled, and how
Our standard crawlers always respect websites’ choices about how their content is accessed and used
Why we care. Crawling is a fundamental requirement for SEO and being found in Google Search and other Google surfaces. This help document might help you quickly understand how Google crawling works and what you can aim to do to improve your site’s crawlability.