Using robots.txt file you can give instructions about their site to web robots. While search engine bot comes to your blog they have limited resources to crawl. So after certain crawling they stop and you website may not indexed. Sometimes there may be some folder or pages you do not want to crawl. For example […]
