Home » Online SEO Tools
Online SEO Tools : Robots.txt
The robots.txt is a very important file to include on your website. It tells the search engine crawlers/spiders/bots what files, pages or directories they are allowed/not allowed to spider. This is important as you might not want them to crawl an image directory and this in turn can save you bandwidth. It also stores the location of your sitemap file. These tools will help you construct and check your file.
Getting your robots.txt file right can be a little confusing. This tool helps you set up which user agents you want to allow and want you want to exclude. Also declares the location of your sitemap (if you have one).
You can use this tool to check if your website or specific pages allow access to robots such as Google,Yahoo,MSN etc.