Every day, hundreds of web crawlers (also known as spiders, bots or robots) will search the web and will most like crawl your website as well. It could be Google trying to index your website to show in their search results, or a spam bot trying to find email addresses to send junk mail to.<\/p>\n
For most webmasters, it is a very good idea to control which parts of your site they are allowed to crawl and visit – and which they are not. To do this, we use a simple text file called By placing There are a number of very good reasons why you might want to restrict access to your site for crawlers by using the robots.txt file.<\/p>\n This can be a big drain on your server resources and it can sometimes cause your server to crash, resulting in down time for your website. It can be useful to monitor your server logs to find crawlers that slow your website down or that spider your site in large volume to block them with robots.txt.<\/li>\n Having these in folders which are blocked with the robots.txt file can reduce the bandwidth and save you an expensive hosting bill.<\/li>\n To do this you'll need to find the name of the bot as it appears in your server log so we can use it to target and blog with robots.txt.<\/li>\n<\/ul>\n It's really easy to get started with a robots.txt file. It's just a plain text file that you keep in the main directory of your site. However, you'll need to be careful to make sure you don't inadvertently block something important. You can easily block your entire site from Google and see your traffic from their search disappear!<\/p>\n To start with, create a plain text document and save it as robots.txt. It needs to be uploaded to the root folder ( Each entry in the file will have a User-agent<\/strong> line to identify the spider you want to instruct followed by one or more Disallow:<\/strong> lines to tell that crawler what to avoid.<\/p>\n So with that in mind let's start with a basic entry into your robots.txt:<\/p>\n In this sample robots.txt the asterisks (*) is a wildcard which targets all User-agents. This is instructing them to not index any<\/strong> of your pages. Don't use this unless you want to completely block Google and all other search engines and spiders.<\/p>\n It's useful to use this if you are targeting a specific agent, such as a spam bot or other crawler you don't want on your site.<\/p>\n Next, you can block specific directories from being indexed by any bots. This is useful if you have admin areas, password protected areas or testing sections of your site which you don't want the search engines to see:<\/p>\n Being specific in your robots.txt will override a previous rule. So if, for example, you have blocked your pages from being indexed by all spiders but you do want to give Google access, you can do this:<\/p>\n Instead of disallowing all of your pages from being crawled, you are also able to allow all of them to be accessed by not putting anything after Some web crawlers (like Google) also accept This is the recommended way for you to set this up according to Google's FAQ page<\/a>, but the robots.txt protocol doesn't officially support it – so only use it for Google at this time.<\/p>\n When you first start creating your robots.txt file, there are a few common mistakes you might make:<\/p>\n A beginners guide to robots.txt and how to use it to control and block spiders or web crawlers from accessing your files and folders.<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":""},"categories":[21],"tags":[],"_links":{"self":[{"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/posts\/1096"}],"collection":[{"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/comments?post=1096"}],"version-history":[{"count":0,"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/posts\/1096\/revisions"}],"wp:attachment":[{"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/media?parent=1096"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/categories?post=1096"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sitebeginner.com\/wp-json\/wp\/v2\/tags?post=1096"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}robots.txt<\/code>.<\/p>\nrobots.txt<\/code> in the main directory of your website (for example: http:\/\/sitebeginner.com\/robots.txt<\/code>) you can advise the crawlers which pages and directories they should access or ignore. Robots.txt is purely advisory though – the search engine spiders can ignore it and still index your site. To avoid this, you can block spiders using .htaccess<\/a> instead.<\/p>\nWhy you should have a robots.txt file for your website<\/h4>\n
\n
Setting up a robots.txt file<\/h4>\n
mysite.com\/robots.txt<\/code> and not mysite.com\/folder\/robots.txt<\/code>). By placing it in the root, search engines and other crawlers know exactly where to look before they begin to index your site.<\/p>\nRobots.txt example 1<\/h5>\n
\r\nUser-agent: *\r\nDisallow: \/\r\n<\/pre>\n
Robots.txt example 2<\/h5>\n
\r\nUser-agent: *\r\nDisallow: \/cgi-bin\/\r\nDisallow: \/admin-area\/\r\nDisallow: \/testing\/test.htm\r\n<\/pre>\n
Robots.txt example 3<\/h5>\n
\r\nUser-agent: *\r\nDisallow: \/\r\nUser-agent: Googlebot\r\nDisallow: \/cgi-bin\/\r\nDisallow: \/testing\/test.htm\r\n<\/pre>\n
Robots.txt example 4<\/h5>\n
Disallow:<\/code>. For example:<\/p>\n\r\nUser-agent: *\r\nDisallow: \/\r\nUser-agent: Googlebot\r\nDisallow:\r\n<\/pre>\n
Robots.txt example 5<\/h5>\n
Allow:<\/code> as a wait to explicitly tell them which pages and directories to crawl. Here, we are disallowing all crawlers except<\/strong> Google:<\/p>\n\r\nUser-agent: *\r\nDisallow: \/\r\nUser-agent: Googlebot\r\nAllow: \/\r\n<\/pre>\n
Common mistakes when using robots.txt<\/h4>\n
\n
Disallow: \/<\/code> which would block the specified user agent from accessing any of your content. This is particularly dangerous with search engine spiders and they may remove your site from their index. Double check to make sure you are blocking the right files for the right bots.<\/li>\nyoursite.com\/robots.txt<\/code> so don't list any files or folders you don't want people or robots to find. Listing the folders in robots.txt will not block them from being accessed directly in the browser or by a bot that ignores the file. If you need to restrict access, use .htaccess to password protect the directory.<\/li>\nDisallow:<\/code> line.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"