Bot

Name: – Bot

Referred to as: Bot, Robot, Spider, Crawler

Category: General, Technical SEO

Updated On: 2021-11-15 by Trevor Stolber

Bot, Search Engine Robot, spider or crawler
Bot, Search Engine Robot, spider or crawler

Description: A search engine robot or “bot” for short is the program that crawls the web. This is also often referred to as a spider or crawler because it crawls the web.

Search engine bots are generally welcomed and the Robots.txt protocol was designed to tell search engine robots what they could and couldn’t crawl on a site.

There are lots of other types of bots which are just automated programs, usually, that scour the web. These can become an unwanted nuisance and consume bandwidth or just gather the information you don’t want from your site.

Google’s search engine bot is called Googlebot and you can see when it comes up in your server logs.

Correct Use: No specific correct use or actions on this topic.

Our take: Search engine robots or bots are your friend, help them find where they need to go with correct robots.txt and meta robots declarations on your site.

Do: Use robots.txt and meta robots declarations

Don’t: Leave meta robots.txt blank

Tip: Make sure that you test your pages are accessible to Googlebot

Subscribe for more SEO Jargon and SEO Tips

Sign up for our newsletter and stay up to date

*

Introduced:

Intro way back when Google was just a little baby, this one is fundamental so the bot age is pretty much the search engine age.

What Google Says:

https://developers.google.com/search/docs/advanced/crawling/googlebot

What experts say: Seeking expert comments