Hi there,

I am in the process of harvesting a large quantity of links to boost PR. I have started putting all my unrelated outbound links on a page which i have disallowed from being spidered with robots.txt.

Apart from annoying a few webmasters who are looking for indexed links, can anyone see any problems that this may cause me? I'm not sure how much attention google actually pays to the robots.txt.

Thanks,

G

Well, to be honest with you that really peeves me off when I get a link exchange with my link on a page like that. I could care less about the PR, I want traffic, if you put me on a page with 0 traffic what good is the link? As far as the SE"s go, you shouldnt have any problems at all, but as far as personal values, its a pretty low thing to do. JMHO

but as far as personal values, its a pretty low thing to do

Your links page will also not get any PR and so it will not interest more people like you who goes for high PR backlinks.

I have a links page with PR which i save for the good sites. This is just for the sites that i get in unrelated catergories.

The visitor can still browse to the links page and it isnt hidden, i have merely told the search engine spiders not to index it in a hope that it will not affect my relevancy rating to my chosen catergory.

I am however dubious that the googlebot actually pays attention to the robots.txt or even if it discriminates against sites that use it.

Anyone have any ideas?

G

Hi Guerrilla,

Googlebot doesn't discriminate sites that use robots.txt to determine whish pages not to be indexed. In Google Information for Webmasters is said: "Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. It is certain that at Googleplex they understand that you may have pages that includes information that shouldn't be available to everyone’s eyes, and therefore there is no reason for them to “discriminate site" for exclusion of that page in robots.txt file - they will just skip that page. IMO you shouldn't do this because it isn't fair to webmasters you are exchanging links with. So my advice is: build categorized directory of relevant sites and those irrelevant you should keep in misc. category – this shouldn’t “hurt much”. Hope this would help a bit

OK, ive removed my robots.txt. I figure that if i get bad rankings i can add it back in for a week at a later date to see it it effects anything.

Thanks,

G

I think that djura has the best idea. Just make your own mini-directory to use for links.