robots.txt is needed if you would like to stop search engines indexing a certain part of your site, assuming that they conform to the rules of course..
That thing is pretty cool, showed me some things I didn't know that spiders could see about my sites, and also showed me things I thought they were seeing and not :-P
__________________
Andrew Rouhafzai, owner of Please login or register to view this content. Registration is FREE
Read the Please login or register to view this content. Registration is FREE of a 17 year old entrepreneur.
That thing is pretty cool, showed me some things I didn't know that spiders could see about my sites, and also showed me things I thought they were seeing and not :-P
well, it seems to me that the tool is a bit to strict.. it isn't showing navigation of sites that I know for sure get indexed perfectly....