Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

Website and Server Administration Forum


You are currently viewing our Website and Server Administration Forum as a guest. Please register to participate.
Login



Reply
Is my Robots.txt not working?
Old 03-16-2009, 11:59 AM Is my Robots.txt not working?
Ultra Talker

Posts: 316
Trades: 0
I have a robots.txt file which I thought would prevent a file from getting indexed/seen by Google but when I type site:domain.org.uk I see this:

http://www.domain.org.uk/_link.php?linkid=bennetts

The robots.txt file is:
User-agent: *
Disallow: _link.php


Is something wrong?

Thanks.
Joe3000 is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 03-16-2009, 12:03 PM Re: Is my Robots.txt not working?
chrishirst's Avatar
Missing! presumed drunk.

Posts: 41,517
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
Nope.

All normal. robots.txt does NOT tell the bots that the URI should not be listed, it simply indicates that it is NOT allowed to index it.

If it finds a link to the URI it will list it as a "PIP" (Partially Indexed Page)
__________________
Chris. ->> Links are advertising NOT optimising!! <<-
A foolish consistency is the hobgoblin of little minds
Thought for today:- I SEO the only industry where all the cowboys are Indians?
chrishirst is offline
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 03-16-2009, 05:42 PM Re: Is my Robots.txt not working?
Skilled Talker

Posts: 77
Trades: 0
To take what chrishirst said even further -- you don't even have to have a link to the page that you don't want showing up in the SERP -- simply visiting the page can lead to its discovery by certain search engines. I know Alexa discovers, and subsequently indexes pages, through toolbar data, and I am mostly certain that Google does as well. I know that I've had googlebot hits to "hidden" pages, but I didn't really follow up with checking whether these pages were indexed or not.

If it's something that you really want hidden, you'll likely have to limit access by setting a cookie, enabling server-side authorization or using a session ID of some type.
__________________

Please login or register to view this content. Registration is FREE

Please login or register to view this content. Registration is FREE
whooligan is offline
Reply With Quote
View Public Profile
 
Reply     « Reply to Is my Robots.txt not working?
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 1.44149 seconds with 12 queries