What is Robots.txt file? | How it is created?

What is Robots.txt file? | How it is created?

(robots.txt-file
Robots.txt is a text file webmasters create to instruct web robots typically search engine robots) how to crawl or fetch pages on their website.

  • The Robot Exclusion Standard is also known as the Robots Exclusion Protocol or Robots.txt protocol.
  • You need a Robots.txt file only if your site includes content that you don’t want search engines to index like Your site Admin Panel.
  • If you don’t want to index Admin Panel then follow this:

User-agent: *

Disallow:/Admin Panel/

  • Link- www.domain_name.com/robots.txt (and save in your directory)

There is an alternative method if you don’t want search engines to index your site:

<metaname=”robots” content=”nofollow”><metaname=”googlebot” content=”noindex”>

Save this command to your HEAD section where you don’t want to index

 

Leave a Reply

Your email address will not be published. Required fields are marked *