The Other Search Engines

**FreeVideoCodes** · 03-29-2005, 03:57 AM

Hello,

I was just viewing my awstats for my website Music Video Codes and I noticed that in the past hour the Inktomi Slurp robot has crawled my website almost 200 times, and every 5 minutes it seems to hit my site again.

My website just went online last week, and is already receiving over 8,000 page views per day. I am also a member of Overture Site Match, but that program only indexes your site every 48 hours.

Does anyone know why suddenly in one day a robot will hit your website soo many times in a few hours?

**kline11** · 03-29-2005, 06:08 PM

Your site might require a deep crawl if you have a large site with several directories.

**FreeVideoCodes** · 04-03-2005, 08:18 PM

Humm... my website only has about 30 pages which are only 30K in size or less

**kline11** · 04-04-2005, 06:47 PM

Are you refering to your site "Music Video Codes"? After viewing the source code, I can see why any bot would have trouble indexing the content (links to DLL's, etc.).

**FreeVideoCodes** · 04-04-2005, 08:24 PM

Hello,

Can you please explain it a little more? I just paid good money for someone to help me code my website, and if it's not SE friendly then I'm in the hole.

Yahoo seems to have only indexed my homepage while Google and Msn have indexed my entire site.

Is there a way i can fix this, or is my entire website wacked up?

**oalhajjar** · 04-05-2005, 02:03 AM

Crawlers are different in behaviour. One day they're all over your site, then you might not see them for a week.