Using robots.txt file you can give instructions about their site to web robots.
While search engine bot comes to your blog they have limited resources to crawl. So after certain crawling they stop and you website may not indexed. Sometimes there may be some folder or pages you do not want to crawl. For example images folder. So using robots.txt you can tell search engine not to crawl that particular folder as this helps you to deep crawling your inner pages also.
Robots.txt file placed at the root of your domain.
www.example.com/robots.txt.
User-agent: * Disallow: /
I basically prevent crawling some folder of your domain but it does not prevent indexing.
– To stop crawling part of your blog use Robots.txt file.
– Do not use Robots.txt file for noindexing rather than No-index meta tag.
For any types of queries, you can contact us on info[at]namasteui.com.
We've gotten so used to seeing men streetwear joggers, ripped jeans, and sleeveless shirts. Hair…
When it comes to festivals, the options for wedding jewellery are endless. You can go…
Whether it concerns your home or an office building, the state of a property’s windows…
You know that running an environmentally sustainable business is the right thing to do. But…
If you are in a financial crisis , or need to start a new business…
To realize the strategic advantage from the fleet management system (FMS) and differentiate the business…
View Comments
Thanks for this post – I followed this for update my not-found urls.