Google has now added new particulars that specify the three classes its Google crawlers fall into, they embody Googlebot, special-case crawlers and user-triggered fetchers.
As well as, Google now lists a JSON formatted file containing the listing of IP addresses every of those totally different crawler sorts use.
Forms of Google crawlers. On the high of this Googlebot page, Google listed these three crawler sorts:
- Googlebot – The primary crawler for Google’s search merchandise. Google says this crawler at all times respects robots.txt guidelines.
- Particular-case crawlers – Crawlers that carry out particular features (similar to AdsBot), which can or could not respect robots.txt guidelines.
- Consumer-triggered fetchers – Instruments and product features the place the end-user triggers a fetch. For instance, Google Website Verifier acts on the request of a person or some Google Search Console instruments will ship Google to fetch the web page based mostly on an motion a person takes.
IP addresses. Google additionally listed the IP deal with ranges and reverse DNS masks for every sort:
What’s new. Right here is the part of the web page that was up to date; the remainder of the web page is generally unchanged.
Why we care. I imagine Google made this variation after they noticed a few of the reactions to the GoogleOther robot they introduced the opposite day. This now explains how Google crawlers act, after they respect the robots.txt and learn how to establish them higher.
Now, if you need to not block Google’s important crawler, Googlebot, however you resolve to dam the others, you’ll be able to higher establish these crawlers extra precisely.
