i have a website that i added to google that has a lot of pages
somewhere aroud 1.100.000 pages
... and have been watching the number of pages indexed by google.
in the webmaster tools i saw that the crawl rate for this website was around 2.000 pages per day ...
and last week it showed 19.000 pages indexed when i did a search for "site:www.mywebsite.com" on google

and now i get 15.000 when i do the same search

Can any of you guys explain why that happened ?

Thanks a lot.

1/ duplicates or near duplicates being removed.

2/ pages that are not "important" being removed.

3/ different datacentres

almost all the pages ... execept let's say 20 have the same overal look
they are pages of company's and have the same design ... ecept for the data ... that is diferent ...
eg : company name, company adress ....
can this be consider similar... and be removed ?

I have been noticing the same type of thing.

the look of design doesn't matter in the slightest.

What matters is the text content of the page.

If the text that is common to all the pages outweighs the unique portion of each of the pages, it is very likely that they will be filtered out.

here are 2 examples
http://www.cazier-firme.ro/caz_firme...php?idd=573018
http://www.cazier-firme.ro/caz_firme...php?idd=835932

Can you tell me if they are 2 too similar ?
Thank you.

The count means nothing. It's an approximation. Quite frankly, I wish they'd just remove the **** thing...it creates too much confusion.

I'm not Romanian, but it seems rather odd that you would have a site with 1,100,000 pages and be concerned enough about SEO to post a question in here. Usually sites of that size are corporate-sized projects, with an entire team working on them. SEO in those situations is priority #912, and almost never gets addressed.

Are you scraping content?

the website is a company directory type ... it has ~ 1.100.000 companys from Romania in it's database and growing ....
We are a young company of coders that just started learning SEO almost a year ago

Originally Posted by solomongaby

here are 2 examples
http://www.cazier-firme.ro/caz_firme...php?idd=573018
http://www.cazier-firme.ro/caz_firme...php?idd=835932

Can you tell me if they are 2 too similar ?

Very possibly. But the first thing that comes to mind is that there aren't 1.1 m pages, there are 1.1 m records, that all get served through the same page. Google recognizes that these are two different URLs because of the query string, but they also know it's the same page.

Also, it looks like a lot of the text on your page is static. Or, you're filling out a form that has labels that don't change regardless of the ID. I do something similar with my photos ( see the text toward the bottom of the pages: Kerry Park, Seattle and Public Market Star Trails ), so I try to add a fair amount of descriptive text about the photo, the location it was shot, or basically something that will (1) entertain curious visitors, and (2) stand out as unique in Google's eyes. I've also created a different page for each URL, so none use query strings.

Then again, it's possible this is just a question of volume.

the fields differ a little form one page to another .... each company has ~ 15 fields but the page only displays the owns that have value ....
Plus each company will have coments from other company's but at the moment a few have so in the future the pages will look very diferent ... but at the moment they seem pretty similar

"Seem pretty similar" is something of a major understatement.

Instinct & experience tells me the site has a major problem.

registration date is October 2006, so even allowing for the aging delay, the pages should not be showing the symptoms they are.

The site has the same issues as several high profile directories had a couple of years ago at launch, one has never recovered from it.

Far too many useless pages with little or no unique content and nothing of value to support that scale infrastructure.

so both 1 & 2 from my first post apply.

well how can this be fixed ... because the pages are not useless ...
They are the pages of diferent company's

I would add a good search tool to your site so that when somebody gets there, they can use your search to find the company they need.

i already have that ....
But first they have to get to my website
If google thinks that my pages are verry similar ... and removes them from it's crawler .. then there goes my content

because the pages are not useless ...

They are for the reasons SEs want to list pages.

They are the pages of diferent company's

Maybe to you, but to the SEs it's 1,000,000 + pages of madlib spam. You need look at why all these pages exist with so little content.

well how can this be fixed ...

Some suggestions:
What's the purpose of the site? When you know this you can set out to make the site fulfil this.

Improve the site navigation. I don't read Romanian, but to have over 1m "pages" and 10 - 15 links or so in the site nav is a little unusual

Originally Posted by solomongaby

well how can this be fixed ... because the pages are not useless ...
They are the pages of diferent company's

How this can be fixed, well, you could offer to buy Google a server and some bandwidth so they can host a million pages in their distributed index without it costing them anything. They don't owe anybody a listing, let alone millions of them. Google's aim is to return good data when people query. Stuff that answers the question. Otherwise people will turn to MSN. So they only keep an index of the pages they deem useful, and if one page is a copy of another, well, only one copy is useful, and it's first come first served.

Don't you have any real content, though? It sounds like you're essentially a directory. Think of other web directories - why would people want to come to your site about a company when they could just go directory to that company's page? How many web directories does Google index "content" from?

well i have content that peoplecould be interested on ...
for each company you can see it acounting numbers from last year like :
- profit
- income
- expenses
- number of employes

but that info also looks similar to google because the thing that is diferent is a number

Okay, what about the other content? The stuff that's on a page besides the one you linked to above? You'll need to SEO that, then when people find you for your other content, they can navigate around your site until they find this.

Gabe I have to agree with much of what's been said here. Even though the pages are unique in the sense that they're showing data from different companies, there's not really a lot on the page. The content is pretty thin as far as a search engine is concerned.

One thing you might try is to create more friendly URLs. Instead of the idd parameter at the end you might use something more unique to the page itself.

I see you're creating unique page titles. I think you might want to create unique meta descriptions too. The rest of the meta tags you're using are essentially useless so you can get rid of them if you want.

Another reason why the pages are being dropped from the index could be that it's low on incoming links in relation to the number of pages the site has. It could also be how quickly the new pages are being added to the site. Both might be setting off some kind of spam filter.

I'm not saying your site is spam, because I think it's completely legit, but it might be showing similar characteristics to a spam site and getting flagged in some way for it. It's a lot of pages to generate in six months and usually it's only spammers that grow that fast.

You might also have to consider that these pages are always going to have trouble staying in the index and ranking. In a sense all they really should be ranking for is the name of the business listed. You might want to create more content heavy pages that can more easily gain visibility in search results and from those pages direct people to the thinner content pages.

Maybe drive traffic to some content rich pages and then use internal search to get people to the business listings.

thank you all for you're answers