Definition of a Robots.txt

Written by Nick Stamoulis

Robots.txt, also known as the robots exclusion standard, is a text file in the root directory of a website. It is a standard that is used by websites to communicate with web crawlers and other web robots. A web crawler is an Internet bot that systematically browses the web. A web robot is a software app that run automated tasks, such as scripts, over the Internet. The robots.txt file tells the crawlers which areas of a website to web and which ones to leave alone. A robots.txt file is often used by search engines to categorize and archive web pages. Webmasters can also use it to proofread source codes.

How exactly does robots.txt work? First, it helps to understand the jobs of a search engine. A search engine crawls the web to discover content and it will index found content so that searchers can find it. In order to crawl sites, the search engine will follow one link to another link and so on, across millions and billions of websites. This is known as “spidering.” But before the crawler can actually spider, it will read the robots.txt file to find out how. If there are no instructions for the crawler, it will crawl other information.

Websites that contain sub-domains usually need a robots.txt file. This is to prevent information that cannot be seen by the public from being picked up for a keyword. Each sub domain on a root domain requires separate robots.txt files.

Before you begin working with robots.txt files, it’s important to know what you’re doing because doing it incorrectly can harm your website. The name is case sensitive, so make sure it is input correctly, without any capital letters.

Where does a robots.txt file go? A robots.txt file will always be places at the root of your domain:

What are the cons of using a robots.txt file? Robots.txt files do have limits. For example, robots.txt directives may not be supported by all search engines. What this means is that while Googlebot and other quality web crawlers will obey the instructions, others may not. In the same effect, different crawlers will interpret syntax differently. It is a good idea to know the proper syntax for each web crawler so that they are all doing the same thing.

What are the pros of using robots.txt files? Because each search spider arrives to a website knowing how many pages it is allowed to crawl, this will help you budget. This is called “crawl budget” by SEOs. It can be worth it to block search engines from crawling problem areas of your website so they can focus on the areas that matter. When you block these areas, it allows you time to go in and fix whatever is necessary to make improvements before letting the crawlers back in.

Robots.txt can also be beneficial for keeping video, audio, and image files from appearing in search results. Individuals will still be able to link to your video, audio, and image files though.

Definition of Robots.txt Sources:

Source #1

Source #2

Categorized in:


Watch How Brick Marketing Can Help Your Company:

Brick Marketing, a Boston-based digital marketing agency, specializes in SEO management, content marketing, social media management, digital ad management (PPC), email newsletter marketing, website development and more.

Contact Us for More Info

Digital Marketing Solutions Include:

Brands & Clients Brick Marketing Has Helped: