SEO Tycoon

**Xtreme2damax** · 04:25 PM

Hello after being away from here for a while I came back because I am having a possibly major issue with my website and search engine traffic. It seems bots and crawlers are having issues with crawling my website. I have summed up most of it in the quote below.

I apologize for the links as this quote is just a direct copy/paste off of my forum.. I just recently purchased an IPB license a little over a month ago and finally made the move to IPB, but this issue seems to have been going on longer than that.

Quote:

I think something is wrong somewhere or broken that is keeping search engines from being able to properly index the website. We technically should at least be seeing a fair bit more guests on at once even if they don't register given the age of the site now.

As most we get between 10 - 30 guests on at once, sometimes less, half the time it's the result of a crawler, crawling the pages..

I posted this poll to see how many found the site via a search engine, and only one person/member found the site via a search engine...

http://www.xtemu.com/forum/?showtopic=763

I would appreciate if someone familiar with SEO can help check if something is wrong somewhere, check meta keywords and descriptions and other general SEO stuff to help me optimize results more efficiently.

Also the web archive wayback machine should typically be showing more archived pages/cached versions of the forum than it currently is. This has led me to believe that something somewhere is breaking search engine crawlers from the ability to properly or efficiently index the pages.

http://web.archive.org/web/*/xtemu.com

http://web.archive.org/web/*/http://www.xtemu.com/forum

The wiki brings up no matches on the wayback machine..

Something is wrong somewhere and given the date of the last cached pages on the wayback machine, might give an idea what could of happened or went wrong.

As you can see one member total came from a search engine on the poll, barely any content for my website is being archived in the web archive wayback machine with five pages total that have been archived. Barely any guests on the site at all, at most we get between 10 and 100 guests, but half or more of guests are made up of crawlers, unknown crawlers so relatively few guests on at a time.

Also whenever a crawler is on the site, it usually sits on the board index not crawling any of the pages or content at all. Also my sitemap generator script is having trouble crawling the site, the end result of a sitemap is unusually small like only a couple hundred kilobytes which is quite small given the amount of content, topics and posts we have.

The pages themselves total about 7MB. So I have came to a conclusion that something is wrong or broken somewhere that is making it difficult or nearly impossible for search engines to properly index the site/forum, which also hurts search engine results for my site. As to what it is that is causing this problem baffles me and I am unable to fix it myself without an idea on what it could be.

Thanks if anyone can help me.

**vangogh** · 12-17-2008, 06:36 PM

Google is showing about 1900 indexed pages on your site so I don't think you have any issue with being crawled or indexed. The way back machine isn't going to archive all or even most of your pages. And when they do show pages listed it's usually long after others have discovered the site.

I'm not sure why you'd expect a lot of people to find your poll through a search engine. The poll is likely specific and in this case the only word in the poll that could be considered a keyword is Xtemulation. I doubt a lot of people are searching for that.

When it comes to things like xbox emulation it seems as though you rank, but about 30 pages back. I'm guessing you don't have many links pointing back to the site and your competition does. For the most part there's only so much you can do with a forum. I don't know what addons are available for Invasion Board, but you could try using any SEO specific add ons. Mostly those will create search friendly URLs.

You also don't really have a lot of content on the forum yet. Most of your forums only have a few threads at most. You're going to need a lot more content before you're pulling in significant search traffic.

**Xtreme2damax** · 12-17-2008, 07:14 PM

Quote:

Originally Posted by vangogh

Google is showing about 1900 indexed pages on your site so I don't think you have any issue with being crawled or indexed. The way back machine isn't going to archive all or even most of your pages. And when they do show pages listed it's usually long after others have discovered the site.

I'm not sure why you'd expect a lot of people to find your poll through a search engine. The poll is likely specific and in this case the only word in the poll that could be considered a keyword is Xtemulation. I doubt a lot of people are searching for that.

When it comes to things like xbox emulation it seems as though you rank, but about 30 pages back. I'm guessing you don't have many links pointing back to the site and your competition does. For the most part there's only so much you can do with a forum. I don't know what addons are available for Invasion Board, but you could try using any SEO specific add ons. Mostly those will create search friendly URLs.

You also don't really have a lot of content on the forum yet. Most of your forums only have a few threads at most. You're going to need a lot more content before you're pulling in significant search traffic.

I understand, but there's the issues with crawlers idling/sitting on the board index, not indexing any threads/posts/forums and it's content most of the time..

Also there's the issue with the sitemap generator script, it's not properly making a sitemap, the end result sitemap is a couple hundred kb, the total size of my pages are much larger than that.

I purchased a license to the unlimited sitemap generator script from xml-sitemaps hoping the create a decent sitemap, but for some reason it hasn't been working as it was supposed to, and doesn't index as it's supposed to..

**chrishirst** · 12-17-2008, 07:58 PM

Quote:

I understand, but there's the issues with crawlers idling/sitting on the board index, not indexing any threads/posts/forums and it's content most of the time..

Sessions are normally 20 minutes so a 1 second visit from a crawler will "stay" on the "visitor" list for that long

Quote:

it's not properly making a sitemap, the end result sitemap is a couple hundred kb, the total size of my pages are much larger than that

A sitemap is only a list of URIs, NOT the page content as well.

**vangogh** · 12-18-2008, 03:32 PM

Also an XML sitemap is no guarantee of getting the URLs listed index. It's a backup in case you have crawling issues with some of your pages. I've never used one and never had a problem getting a site indexed.

Again I really don't think you have an indexing issue. Your site appears to be indexed fine. Ranking and indexing are two different things. Your pages are indexed, but at the moment your competition is ranking better. Put more focus into gaining more members and creating more threads and posts on the forum. Also see if you can get other sites to link back to yours.

**kidsnsciencee** · 12-23-2008, 03:05 AM

to overcome crawler indexing problem, site should be having good interlinking and having updated the website related work in google webmaster tools so google regular crawl the pages

**vangogh** · 12-23-2008, 08:37 PM

True, but there really isn't an indexing problem here. The problem appears to be more a ranking issue.

**miketyson** · 12-26-2008, 03:10 AM

Correct or remove broken links from the site and make google sitemap that will help in crawling properly.

**~ServerPoint~** · 12-26-2008, 04:09 AM

Quote:

Originally Posted by miketyson

Correct or remove broken links from the site and make google sitemap that will help in crawling properly.

That might help and at the same time that might take a lot of time. Just be aware of that. Google is a bit slow guy.

**witnesstheday** · 12-28-2008, 02:40 PM

Quote:

I posted this poll to see how many found the site via a search engine, and only one person/member found the site via a search engine...

From my experience from watching forums, posting at forums, lurking in forums, abusing forums, occasionally (for a few days) building forums (and then chucking them away since it's not really my field, to be honest) I have worked out that 90% of the users don't interact, they just read.

No doubt even when you have 2000 members and 150,000 posts, a poll will not really capture the real size of search-engine driven traffic.

6000 posts... could surely not raise more than a couple of 100 users per month? Or am I way off? There's gotta be at least 500 forum owners on this forum... maybe someone has some sample data.

**Xtreme2damax** · 03:17 PM

Maybe I wasn't clear enough, crawling is definitely broken, this includes sitemap generating scripts as well. Sitemap scripts don't index all of the URL's it is supposed to. It indexes fairly few URL's and justs quits claiming it completed.

I use the unlimited XML sitemap generater from XML-Sitemaps.com that I paid for a couple months ago.

Ever since the move to IPB, crawling has been broken in some sense one way or another.

Very rarely will a crawler index a post, topic or page, 90% - 100 % of the time the most it'll do is idle on the index and not do anything at all then leave. I observe the crawlers actions on the who's online feature for the duration of the crawlers visit and nearly 95% - 98% of the time it sits on the board index for the duration or all of it's visit.

I can copy a unique phrase from a members post, paste it into Google and Google doesn't bring up any results for the topic/post with the phrase I pasted into Google. My page isn't showing up the way it normally should on search engines for unique keywords and phrases, and is usually way back in Google, Yahoo, MSN.

Crawlers will sometimes index threads/topics, but they don't seem to be indexing the content of the topics/posts and if they are it isn't doing it efficiently.

As far as I know there are no broken links or relatively few broken links on my website or leading to my website.

There is something wrong here when smaller similar websites with less content that are younger are pulling in more traffic and registrations than I am.

I've been working on filling some sections with content including the inactive sections which has made little to no difference yet.

I get relatively few visitors on at a time, about 5 - 15 visitors, and most of the time it's an unidentified crawler.

Just because my website shows up on Google does not mean that the crawlers are indexing it as they should or as efficiently as possible. I have done observations for the past month or so and have concluded that at some point, something is breaking crawling of my website/forum. As to what it is that is breaking crawling, I have been unable to figure out.

**vangogh** · 01-15-2009, 08:02 PM

So is the issue that you're Sitemap script not working or that search engine spiders aren't crawling past your home page? If the former the issue is likely your script. If the latter how are you testing to know that the crawler is having a problem?

Search engines don't always crawl every page of your site. There are shallow crawls and deep crawls. They won't crawl every page of your site every time they visit your site.

Just to check I looked at this URL on your site.

http://www.xtemu.com/forum/index.php?showtopic=895

I then checked Google by searching on the URL. The page was returned as the lone result and I clicked the cache button. Google showed Jan 2, 2009 as the cache date. I then pulled a few unique phrases from the page wrapped quotes around the phrase and again searched Google. Each time Google returned the page as the only result.

If you say there's a crawling problem, then there's a crawling problem. Looking at what I can see from here it doesn't seem to me there's a problem. I just tried one more search:

site:http://www.xtemu.com/forum/

That pulls up a list of your indexed pages on the forum. Checking the cache of a random few the cache date is showing from January. I didn't look through every result returned and I have no idea how many pages you have on the forum so maybe there are many that haven't been indexed. But I see plenty of examples of Google having crawled your forum past the home page and those pages being updated in the index within the last week.

When I see that it's hard to believe you have a general problem being crawled. I admit I could be wrong, but I can only go on what information I can collect. How do you know search engine spiders are not able to crawl your site?

**chrishirst** · 01-15-2009, 08:30 PM

crawlers will seem to "sit" on the index page because of the session timing. Ignore what the forum software is telling you and read your site logs.

crawlers do not crawl. They request individual URIs as dictated by the scheduler/crawler queue in the datacentre they are sent by. The reason that SEs visit your "home" page so much is because that's where the majority of your links point

Your site has 3000+ pages in the Google index. Xenu and WinHTTrack had no problem accessing the pages, Xenu I stopped because of the session variables and WinHTTrack I stopped because I don't really need a copy of your website

The crawler for the "wayback machine" is not very active these days and the archive is about 12 months behind most changes

Quote:

There is something wrong here when smaller similar websites with less content that are younger are pulling in more traffic and registrations than I am.

This is nothing to do with crawling, it's just down to lack of promotion.

**Xtreme2damax** · 01-18-2009, 10:01 PM

Sorry about that then, I guess I get a bit worried about things sometimes and thought there is/was something wrong

Is there anything else you can suggest I do to help improve my ranking on search engines besides just actively posting or actively posting new content?

Are meta's punished by Google and other search engines, or am I safe adding some keywords in my Board Header and Footer Wrapper or using a dynamic meta modification for IPB?

For SEO I just want to know key things that drive traffic to websites, not read an entire bible/article on SEO..

My SEO options are limited are a bit limited, it appears the only free SEO modification for IPB (Minerva SEO) is dead or development has ceased, it's developer has disappeared without an announcement on whether development has ceased or if the project it still alive. Currently the modification has some serious bugs that affect proper operation of the forum, such as page links in muti-page forums and topics redirecting to member profiles, forum index or random threads.

Community SEO for IPB which is still active, just costs to much, I'd like to buy a licence but not for $150, same as the cost as that of IPB, $30 - $60 would be more appropriate.

**chrishirst** · 01-19-2009, 12:44 PM

Quote:

For SEO I just want to know key things that drive traffic to websites, not read an entire bible/article on SEO..

SEO is NOT what drives traffic to sites! SEO is what improves conversions once you get visitors to your pages. SEO is what improves your site in the hope of attracting traffic from search engines results.

SEM, Marketing and Promotion is what really gets more visitors

**full house** · 01-20-2009, 09:03 PM

Quote:

SEO is NOT what drives traffic to sites! SEO is what improves conversions once you get visitors to your pages. SEO is what improves your site in the hope of attracting traffic from search engines results.

SEM, Marketing and Promotion is what really gets more visitors

for me, SEO do all that! they get traffics and also improve the conversion.

**importexportus** · 01-22-2009, 01:11 AM

Accurate
Excluding present in actual fact isn't an indexing trouble at this point.
The difficulties materialize in the direction of subsist extra a ranking problem.

**full house** · 01-22-2009, 09:45 PM

google still indexing your site. problem would be some back links have been removed cause google update the back links.

**Kwinzee** · 07:05 AM

Quote:

Originally Posted by chrishirst

The reason that SEs visit your "home" page so much is because that's where the majority of your links point

I am gonna re structure my site so sub categories have index pages. Which will all point to the relevant content.

So if I allowed the url of the sub category index page in my robots.txt, would this help the indexation?
Or would it be sufficient by adding Allow: /, as all webpages link back to the home page via the header or footer? And the robot finds the inbound links to the home page, from within the internal website structure. Is that kinda what you are saying?

**chrishirst** · 08-21-2010, 12:36 PM

Nope. Absolutely nothing to do with robots.txt at all.

I'm talking about links that actually point to your content.