The standard for the exclusion of spiders uses a simple text file. This file has to be called "robots.txt" and it contains some instructions that can forbid to some or to all spiders the download of a site's pages.
If you have already created a robots.txt file, you have to put it in the main directory of your site. For example if the site is http://www.globalwarming-awareness2007.com, the file will have to be to the address http://www.globalwarming-awareness2007.com/robots.txt .
The robots.txt file contains
records and each record is formed by two
fields, "User-agent" and one or more "Disallow".
User-agent field shows which spider the next directive is referred to.
Disallow field shows which files or directories are forbidden to the spiders specified in User-agent field.
First example:
User-agent: googlebot
Disallow: /secret/
This record forbid googlebot (Google's spider) to download files contained in the folder called "secret".
Second example:
User-agent: *
Disallow:
This record says to spiders that they are allowed to download each site's pages.
This article about robots.txt file could help this site and GT-Team to win
globalwarming awareness SEO contest!