Updated On: 2021-11-15 by Trevor Stolber
Name: – Bot
Referred to as: Bot, Robot, Spider, Crawler
Category: General, Technical SEO
Correct Use: No specific correct use or actions on this topic.
Description: A search engine robot or “bot” for short is a program that crawls the web. This is also often referred to as a spider or crawler because it crawls the web.
Search engine bots are generally welcomed and the Robots.txt protocol was designed to tell search engine robots what they could and couldn’t crawl on a site.
There are lots of other types of bots which are just automated programs, usually, that scour the web. These can become an unwanted nuisance and consume bandwidth or just gather the information you don’t want from your site.
Google’s search engine bot is called Googlebot and you can see when it comes up in your server logs.
Our take: Search engine robots or bots are your friend, help them find where they need to go with correct robots.txt and meta robots declarations on your site.
Do: Use robots.txt and meta robots declarations
Don’t: Leave meta robots.txt blank
Tip: Make sure that you test your pages are accessible to Googlebot
Introduced:
Intro way back when Google was just a little baby, this one is fundamental so the bot age is pretty much the search engine age.
What Google Says:
https://developers.google.com/search/docs/advanced/crawling/googlebot
What experts say: Seeking expert comments