SEO Tycoon

**PrinceOfBH** · 02-05-2007, 12:48 PM

Hi,

I wanted to know how bots handle directories. I know they crawl pages but what about directory. Say for example, we have two directories .. apples and oranges .. now there is a link to a file inside apples directory, so I assume bot will crawl apples directory. But since oranges directory is not listed in any web page, it wont be crawled (or will it be crawled?). I've got some directories on my webserver which are for developing basis and admin section. I didn't want bots to crawl things inside those directories. Is it necessary to put the directory names in the robots.txt file because I wont be linking them to pages?

Thanks in advance

**stOx** · 02-05-2007, 01:27 PM

If a page is linked to, google will find it, if it's not linked to, they wont.
To be safe though you should exclude any files or directories that you don't to be indexed with a robots.txt

**Learning Newbie** · 02-05-2007, 06:23 PM

There are lots of bots on the internet. The search engines all use them for cheap labor, but so do all kinds of other people, companies, hobbysts. You can buy a "bot" to run on your computer and download free music for you, even. So there are too many to say how they all work.

But the search bots work on links, not directory listings. They don't even know what all the files in a directory are.

If you make a new page on your site but don't link to it from anywhere, it will be a secret page. No one will ever find out about it, even if there are other pages in the same directory.