The Other Search Engines

**disco_troll** · 12-15-2004, 09:29 AM

Hi -

Following on from my earlier post regarding my site, selling widgets of all shapes and sizes (and colours!

), the only search engine to bother spidering my entire site is Google (though until it gives the site a PR, that doesn't count for much). The likes of Yahoo, MSN, AllTheWeb, etc have only crawled (or registered) the homepage and one or two others.

The site has text links on it, a link to a central sitemap on every page and the sitemap page has also been submitted to all search engines.

Has anyone any advice on why the majority of search engines haven't registered the entire site?

I have launched some other projects recently (all built the same, basic, way) and it seems the domains that are rebuilds have been crawled entirely within a few days, whilst the completely new sites are lucky if these search engines even bother updating their cached version of the homepage, even if it was a simple ISP holding page.

Cheers,

DT.

**Jamesanderson** · 12-15-2004, 11:12 AM

Hi disco_troll

What type of URL extensions are you using, ie www.url.com/blue-widget.html or www.url.com/?=widget+blue

If it is the later it will take a little longer to get indexed.

**disco_troll** · 12-15-2004, 12:12 PM

Hi James - to my rescue again!

Well I'm not sure how the the search engines would read the pages. They are all in PHP so the route through the site goes like this:

homepage and news, privacy policy, Ts & Cs, etc - all in html - most registered with search engines.

Product pages - all in PHP - all initially load in the browser without the ?-string (at the moment) and include a large amount of static copy and content, though each page might well offer between 5 and 10 additional pages of information pulled from the database. These additional screens do include the ? part of the address and if you navigate back to the first page then it, too, contains the ? part of the address.

On all these pages it is a very simple www.url.com/?=page1 type of dynamic address. Is all this enough to slow down the registration process?

Thanks again.

DT.

**Jamesanderson** · 12-15-2004, 12:45 PM

I would think so, if your producing one page for the search engine then they follow all the links and eventualy come back to the same page but with a string they will class it as two pages but with duplicate content. This could be the reason why it is taking so long.

I would try to change all the URL's to be the same (without string if possible but not essential). Also the other search engines dont crawl as frequent as Google so it may be just be a matter of waiting it out.

**Rincewind** · 12-15-2004, 08:39 PM

If you have a very complex or deap site (one with lots of links to other links to other pages) then the deap pages may not be considdered important by the search engines. Try getting some of you link partners to link directly to the content on the deap pages rather than just your home page. Too often people concentrate all their incoming links on the home page of their site and forget the other pages.

Remember that Page Rank is about Pages not Sites. It's important to get a good link popularity for every page on the site not just the for the site it's self. If you have external links pointing deap down in your site, then the spiders will start deap down and work out rather than starting high up and working down.

Also, there can be cases where you create loose ends on your site. Try drawing your site map. Are there any branches that don't go anywhere? Dead ends? Sometimes adding a link to another site form a location deap down in your site will pull the search engine down to follow that link. Where as, if you leave it as a dead end, the then spider just passes that limb of your site by.

**disco_troll** · 12-16-2004, 09:01 AM

Cool -

My external link sites that I've just created all link to sub-pages so eventually they should all have a good page rank on their own merit, rather than through the homepage only.

The site structure is very shallow with the ability to go directly from one page to any other as I've experinced the 'dead-branch' scenario before and try to avoid it.

As it is purely the PHP pages that are yet to show up I get the feeling that when one goes, they'll all go!

Thanks again for all the advice.

DT

**neorunner** · 01-25-2005, 01:26 PM

You should also consider doing mod_rewrite on your php urls. This would make it easier for the SE's to get to the deeper pages. However, mod_rewrite is a pain in the butt. If your content and backlinks are good enough google will eventually get through many pages. My site has over 500,000 pages, and google only has 16,000 or so indexed. I have considered doing mod rewrite on my site to make it easier for google to find its way around. We shall see!

Good luck.

Neorunner