Google has added a new user agent to its help documentation named Google-CWS. This is the Chrome Web Store user agent that is a user-triggered fetchers.
More details. Google posted about the new user agent over here, it reads; “The Chrome Web Store fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes.”
What are user-triggered fetchers. A user-triggered fetchers are initiated by users to perform a fetching function within a Google product.
The example provided by Google was “Google Site Verifier acts on a user’s request, or a site hosted on Google Cloud (GCP) has a feature that allows the site’s users to retrieve an external RSS feed. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The general technical properties of Google’s crawlers also apply to the user-triggered fetchers.”
Why we care. If you see this user agent in your crawl logs, you now know where it is from. The Chrome Web Store fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes.