SEO Tycoon

**magicvw** · 02-05-2009, 08:07 AM

If I have a password protected directory can the spiders index it? I haven't stopped them in my robots txt, so are they allowed in? thnx

**chrishirst** · 02-05-2009, 08:09 AM

No ....

**magicvw** · 02-05-2009, 08:10 AM

ok thanks!

**full house** · 02-05-2009, 09:34 PM

why no? Just confuse!I wanna know.

**colbyt** · 02-05-2009, 10:01 PM

Quote:

Originally Posted by full house

why no? Just confuse!I wanna know.

Because they don't have the username or password.

**francis84** · 02-06-2009, 02:41 AM

As far as I know, NO. Website with Password Protection make the spiderbot find it hard to crawl (or should I say, NOT TO BE CRAWL) by spiderbot.

**CoolDude55** · 02-10-2009, 12:02 AM

absolutely NO

**vetzkov** · 02-10-2009, 11:10 AM

spiders act like visitors in the way they access a website so it is not possible to crawl a password protected website

**full house** · 02-11-2009, 09:08 PM

are you saying those site with username and password like wordpress, blogger and others?

**Moxxnixx** · 02-11-2009, 10:09 PM

Excuse my ignorance...
Sometimes, when doing a search on Google, I'll click a link and I won't see the page because I have to login to see it...like a forum, for example. How is Google indexing those if they're password protected?

**chrishirst** · 02-12-2009, 09:36 PM

Quote:

Originally Posted by Moxxnixx

Excuse my ignorance...
Sometimes, when doing a search on Google, I'll click a link and I won't see the page because I have to login to see it...like a forum, for example. How is Google indexing those if they're password protected?

There are ways of allowing SE bots into a password protected site.

**magicvw** · 02-13-2009, 03:13 AM

Quote:

Originally Posted by chrishirst

There are ways of allowing SE bots into a password protected site.

How?

I know I cab authenticate adsense to show up on my protected pages, but I guess that's different?

**jhendriana** · 02-14-2009, 12:15 PM

im really believe NO. cause thats need username and passwords, thats why called protected dirs

**chrishirst** · 02-21-2009, 07:46 AM

Quote:

Originally Posted by magicvw

How?

I know I cab authenticate adsense to show up on my protected pages, but I guess that's different?

You give the crawlers you want to allow in a "free pass" by detecting the user agent signature and bypassing the authentication process if it is a match.

**weblover50** · 02-21-2009, 09:24 AM

Mate I haven's understand the thread? What are you talking about?

**zeruel** · 02-24-2009, 10:18 PM

Quote:

Originally Posted by weblover50

Mate I haven's understand the thread? What are you talking about?

lol. just read from the top. It's all english.

oh well, chrishirst hit the right point. That's how the bots crawl the sites. They are not like any other person browsing the website and checking it if they can access it or not. They are bots. Spiders. Far different from a human.