Soon, the Googlebot Crawl Rate Limiter Tool will no longer be available in Search Console and will be removed.
Due to advancements in crawling, which have essentially rendered the crawl rate limiter tool unnecessary, Google announced that it is sunsetting the tool, which was set to be removed on January 8, 2024.
The Crawl Rate Limiter Tool in Search Console
2008 marked the fifteenth anniversary of the crawl rate limiter tool’s introduction to the search console. The tool’s objective was to give publishers the ability to manage Googlebot crawling so that the server wouldn’t become overloaded.
There was a time when excessive crawling was experienced by some publishers, which could cause the server to fail to deliver webpages to users.
Google eventually released the tool within search console after receiving enough complaints.
The tool’s impact was to supply Google with information. Google claims that requests to restrict crawling usually took a day or so to take effect and lasted for ninety days.
Reasons Google Is Eliminating The Rate Limit Tool
According to the announcement, crawling algorithms have advanced to the point where Googlebot can recognize when a server is full and immediately reduce its rate of crawling.
Additionally, Google claimed that the tool was rarely used and that the crawl rate was usually set to the lowest setting when it was.
From now on, the minimum crawl rate will be automatically set to a lower rate, in line with what publishers have typically requested.
As stated in the announcement:
We’re also lowering the minimum crawling speed to a lower rate, similar to the previous crawl rate limits, in conjunction with the deprecation of the crawl limiter tool.
This implies that our crawlers don’t waste the bandwidth of the website and that we essentially keep up with the preferences that certain site owners have previously set if the search interest is low.
Simplifying the Search Console
Because there are fewer rarely used tools cluttering search console, it is easier to use after the tool is removed.
Consequently, this ought to enhance the search console user experience.
Publishers can still provide Google with feedback using the Googlebot report form if they are still experiencing issues with the crawl rate of Googlebot.
