Search | View Active Threads | View New Posts
Current Time: 10:55 AM

 

+ Reply to Thread
Results 1 to 6 of 6
  1. #1

    What is robots.txt?

    What is robots.txt?

  2. #2
    Hi Friends,

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

    Dynamic Web Design Company Bangalore | Joomla Web Development Company Bangalore | Php Web Development Company Bangalore

  3. #3
    Join Date
    Jun 2018
    Location
    surat
    Posts
    586
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

  4. #4
    Join Date
    Feb 2018
    Location
    US
    Posts
    387
    Robots.txt is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  5. #5
    Join Date
    May 2019
    Posts
    72
    A robots.txt file lives at the root of your site. So, for site www.example.com, the robots.txt file lives at www.example.com/robots.txt. robots.txt is a plain text file that follows the Robots Exclusion Standard. A robots.txt file consists of one or more rules. Each rule blocks (or or allows) access for a given crawler to a specified file path in that website.


    Drive Thru Speaker Post | Retail Management System | Cloud Retail POS

  6. #6
    Join Date
    Feb 2019
    Location
    CA
    Posts
    128
    Robots.txt gives indexing instructions to search engine.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  

Sponsors

Domain Escrow Services
Above.com Domain Registrar, Marketplace and Revenue Optimizer
Trellian Direct Search Network

Industry Events

11-14 August 2019
Affiliate Summit East
New York, NY

4-6 March 2019
LeadsCon
Las Vegas

15-18 September 2018
MERGE 2018
Orlando, FL

Poll

How do you invest in domains?
to see the Poll results!
DomainState.com
Domain Tools | Domain Directory | Registrar Stats | Domain Glossary | Industry Events | FAQ | Members | Terms | RSS | Link To Us | Advertise | Contact Us
Other Related Trellian Services:
Above.com Registrar  |  Above.com Marketplace  |  Above.com Domain Manager  |  Free Search Toolbar  |  Free Webpage Builder  |  Keyword Research
Direct Search Navigation  |  Search Engine Submission and SEO Tools
Copyright © 2002 DomainState.com a Trellian Company 2014 Bronze Sponsor - Internet Commerce Association