my site still isnt in the top 1000 in google for the keyword btcc, i've changed the meta data to be unique to the page, i recently put a robots.txt file in my site root, but this still doesnt seem to be helping, the site does show well in MSN however

all my webpages are at the site root, my question is this

do search engine bots only look for pages in folders within my webspace? thus explaining why it doesnt seem to be "seeing" my pages at root level

is the robots.txt correct? should just be blocking all search engine bots from looking in the specified folders
http://btcc-racing.com/robots.txt


cheers in advance

great looking site. my guess is that you are just trying for a pretty hard search term. BTCC shows up on tons of car websites. i probably see that written at least a few times a day when i surf on audiworld.com. you're going to need a lot more backlinks in google, especially with BTCC in your anchor text. good luck, but its gonna take you a bit of time.

cheers!

how do i go about getting "backlinks in google" and what are they?

anyone got any ideas on the other questions in my first post?

its just a see some amateur stuff in the top 20 and i dont understand how its got there, have i done something fundamentally wrong thats hurting my chances?

Check your google page saturation by doing a search for "site:www.btcc-racing.com". This will tell you how many pages google has indexed.

If your website is not returning anywhere in the top 1000 results when you actively target certain keywords, then another likely cause of your problem is that google has "Sandboxed" your domain. You can also confirm by doing the same search at other engines, msn for eg. If your site turns up there and not google, your boxed. It's theorized that the sandox is a filter for new domains to combat linkfarm/network strategists. New domains return high in results for a few weeks after launch, then suddenly drop off into the abyss. This is the sandbox.

New domains generally get boxed for 6-9 months depending on the popularity of the key terms the site targets. This phenomina is is actually good for webmasters. It forces them to learn how to attract traffic via non-google means. Don't bust your brain trying to get into google's good books if your boxed. Optimize your pages without spamming and when the filter is lifted for your domain at google you'll do well.

Your 'backward links' are going great, especially at msn. Try and get as many incomming links to your site as possible from other sites with a hi PR ranking at google. This is the best strat for upping your own domain's PR.

cheers metho

i've had a look and im almost certain that my sites in the sandbox, what you describe is exactly how it is, the few pages the google has indexed are only the ones i had in the first few weeks the site was launched,

i'n the meantime i'll try and get more links on my site from the high ranking ones in google, google is what 99 percent of people use isnt it?

is there a free app for simultaneously searching for keyword ranking positions on many search engines at once?

i've had a good search on many search engines best results (for 'btcc') are msn 19th, exactseek 32nd, lycos 109th, teoma and askjeeves 11th page of results

im guessing that my robots.txt is fine

cheers

There have been some weird issues with robots.txt and GG later - there is another well know SE forum that talks about it in detail. Hope that helps

cheers for the the reliable and precise info guys,

near enough 6 months later et viola practically my entire site seems to have been indexed in google - thanks!

good to hear man.

Robots.txt shouldn't affect your rank at all. It's just used to disallow access to certain resources - for example, disallowing access to your images folder so Google doesn't make your graphics available to Joe Public.