Create Robots.txt File - Robots Text File
Creating a robots.txt file
Frequently Asked Questions
Creating a robots.txt file will not improve your search engine
positioning, but it does provide robots with information concerning
which files you will not allow to be crawled and indexed in the search
engines.
When a robot crawls your site it looks for the robots.txt file. If
it doesn't find one it assumes automatically that it may crawl and
index the entire site. Not having a robots.txt file can also create
unnecessary 404 errors in your server logs, making it more difficult
to track "real" 404 errors.
Assuming you want your entire site indexed and only want to stop
the unnecessary 404 errors from occurring you have a couple of
options.
back to top
Please note: This will allow all robots to crawl and index all files.
This allows all robots to crawl all files.
User-agent: *
Disallow:
back to top
Please note: Disallowing a specific file to be crawled will keep it from
being indexed. The file disallowed will not show up in the search
engines. HOWEVER, this is only effective for friendly robots. Robots
can choose to ignore your instructions.
This allows all robots to crawl all files except the images file.
User-agent: *
Disallow: /images/
This allows all robots to crawl all files except the images file
and the stats file.
User-agent: *
Disallow: /images/
Disallow: /stats/
back to top
Occasionally you may find that you would like to disallow specific
robots from crawling your site or limit which files they may have
access to.
This denies access to Googlebot-image to any files in your domain
User-agent: Googlebot-Image
Disallow: /
This specifically denies Googlebot-image to your images file
User-agent: Googlebot-Image
Disallow: /images/
For a current data base of robot names and information, visit:
http://www.robotstxt.org/wc/active/html/index.html
back to top
Simply create a text document and save the new document as
robots.txt Do not use a html editor to create the file unless is has
the ability to create a plain text document (ASCII). Most computers will allow you to create a text document using
notepad.
-
Right click on your desktop
-
Choose new
-
Choose text document
-
Open the document you just created
-
Insert instructions to robots
-
Click on save as
-
Save document as robots.txt
back to top
Once you have uploaded the file to the root directory of your
domain it's good idea to use a robots.txt validator to confirm that
everything is correct. You can search Google for
free robots.txt validator or try the one listed below.
http://tool.motoricerca.info/robots-checker.phtml
back
to top
This page is intended to cover creating a very simple robots.txt
file. If you require a more detailed robots.txt file for your website
there are many help resources available on the net. Google Information
for Webmasters recommends visiting:
http://www.robotstxt.org/wc/norobots.html
J. Walker of GNC Web Creations
Contact Me With Your Comments or Questions
Articles are Copyright © J. Walker of GNC Web Creations Unless Otherwise Noted
We welcome your comments on creating a robots.txt file.
Comments and Suggestions
Additional Search Engine Optimization Article
Search Engine Optimization Articles
Ethical Search Engine Optimization Class
SEO Techniques - Search Engine Optimization
You've ended up in the right place to learn the basics of SEO, along with many valuable internet marketing strategies,
designed to increase your online business profitability and visibility
in the search engines. We provide structured online classes through our Yahoo group
SEO Techniques. The online classes are 100% free and are email based so you are able to work at a pace that best fits your needs.
Ethical Search Engine Optimization Class
Information on Ethical Search Engine Optimization Class
SEO Techniques Step by Step eBook
Search Engine Optimization Techniques
When the website challenge you are facing is poor search engine
placement, consider SEO Techniques Step by Step as a low-cost alternative
to SEO consultant fees.
 |
Many members have found class much easier to follow, and have
consistently reported higher success rates in the search engines by owning the eBook,
SEO Techniques Step By Step. It is not a requirement of
attending class. You will be entirely welcome either way!
We're committed to doing our very best to exceed your expectations! If you have
questions or comments, please don't hesitate to
contact us.
|
SEO Techniques Step By Step
The eBook With All The Lessons Compiled Here
Sometimes a program is so incredible that we have to
share it with others! We are extremely impressed with this all new
version of
Xara Webstyle 4 and are currently using it as a tool in designing
our sites! This is a "must have" tool for web designers! Take a look at
our complete
Xara Webstyle 4 Review today!
Try Xara Webstyle 4 FREE Today!
GNC Web Creations - 2125 Universal City BLVD - Universal City, TX 78148 - (210) 564-9153
Copyright © 2003 - 2011 J. Walker of GNC Web Creations All Rights Reserved
| 
