Updated on: 2022-02-23
Name: robots.txt
Referred to as: n/a
Category: On Page SEO
Description:
Robots.txt is a file that indicates to search engine bots which pages they should and should not crawl on website.
Every major search engine (like: Google, Bing and Yahoo) recognizes and follow Robots.txt requests. It should be noted however that the robots.txt protocol is deprecated and is now a hint, not a directive. For full control you should use either the meta robots tag or the X robots header response code. That being said, it is still considered SEO best practice to use a robots.txt file.
Correct Use:
The correct format of robots.txt :
User-agent: X
Disallow: YUser-agent refers to a specific bot or robot that you want to stop crawling.
It is possible to block anything (pages or sections or folders) using the “disallow” attribute.
For example:
User-agent: googlebot
Disallow: www.example.com/about-usIt tells Googlebot not to index this url or page of your website.
You can also use an asterisk (*) to define any even all bots that stop by your website.
For example:
User-agent: *
Disallow: /birdsThe “*” tells any and all spiders to NOT crawl your birds folder.
There are many more rules also included. You may visit google’s blog to find the rest of them.
Our take:
The main objective to create a robots.txt file is proper crawling guidelines for search engine crawlers. So that search engine bots do not crawl certain pages or parts or files of a website.
A robots.txt file is recognize to maintain crawler traffic to your website, and usually to keep a pages or file off Google, varying on the file type:
Web page: Robots.txt file generally used for web pages (HTML, PDF, or other non-media formats that Google can read.
Media file: It is used to prevent different media files like image, video, and audio files from appearing in Google search results.
Resource file: Robots.txt file is also used to block resource files such as unimportant images, personal info, script, or style files.
Extra reading:
- Google Blog
- How to interpret the robots.txt specification
- Robots txt deprecated by Google
- Introduction of Robots.txt in 1994
Do:
Make sure to disallow your personal information as well as a resource file. It will help you to block any other bots.
Don’t:
Before creating a robots.txt file, you must be aware of all the rules.
Tip:
As robots.txt file is case sensitive, make sure to use a lowercase in the filename.
Tip: As robots.txt file is case sensitive, make sure to use a lowercase in the filename. Click To TweetIntroduced: 1994
What Google Says: How to create robots.txt