The Google Forum

**stevefrm** · 12-21-2009, 02:39 PM

My site is not get crawled for very long and when i checked it on webmaster tools it shows
"Truncated response"

for my web pages.

Please let me know what to do so that it get crawl and index new edited page.

here is my site-
"http://www.karyonconsulting.com"

**chrishirst** · 12-21-2009, 02:45 PM

Maybe it was having a bad day!

BTW you need to fix this

HTML Code:

<!DOCTYPE html>

it's malformed and may be causing the indexer to fail. though unlikely

**stevefrm** · 12-21-2009, 04:02 PM

But when you check for google.com source code it is the same
<!doctype html>

Can it create problem in crawling??

**chrishirst** · 12-21-2009, 04:11 PM

Does Google need to be crawled??????

**stevefrm** · 12-21-2009, 04:52 PM

Please do not be angry but may be I am asking stupid thing
i checked for google and it shows as-

"This is Google's cache of http://www.google.com/. It is a snapshot of the page as it appeared on 21 Dec 2009 05:57:53 GMT. The current page could have changed in the meantime. Learn more"

So in this way if google can be crawled with the same "<!doctype html>" in source code then why not my site.

I know Google is Google - the big thing but if that have same in source code as my site so why not my site cawled and giving error for this?

**chrishirst** · 12-21-2009, 05:08 PM

It may or may not be a problem, it is merely a possibilty.

If you are happy that browsers may not be rendering your pages correctly, then leave it as it is.

If you are happy that SE indexers may not be reading your pages correctly, then leave it as it is.

Of course it may come as a surprise to you to find that Google DO NOT crawl their own pages (blocked in robots.txt) (why should they crawl them?) so they are NOT indexed or analysed by the same mechanisms, and why should they be??

THEIR pages are not going to appear in the results pages, while you want yours to be there.
CONSEQUENTLY the malformed code will NEVER be a problem for Google's pages (if that is the problem of course).

But hey! it's your site of course and your decision ultimately.
We can only advise.

**stevefrm** · 12-21-2009, 05:18 PM

Ok thanks a lot but now please tell me one more thing that what should I put in place of-

"<!doctype html>"

in source code of my site.

**chrishirst** · 12-21-2009, 05:20 PM

a correct and valid DTD

http://www.w3.org/QA/2002/04/valid-dtd-list.html

**stevefrm** · 12-21-2009, 05:24 PM

Thanks but I am confused which one to use for my site.

**chrishirst** · 12-21-2009, 05:32 PM

HTML4.01 strict I would suggest.

**stevefrm** · 12-21-2009, 05:39 PM

Should I add all 6 lines mentioned under HTML4.01 strict

Quote:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN"
"http://www.w3.org/TR/html4/frameset.dtd">

**chrishirst** · 12-21-2009, 05:42 PM

the top two lines are the Strict DTD

the next two lines are the Transitional DTD

and the last two lines are the Frameset DTD

**chrishirst** · 12-21-2009, 05:44 PM

The top two lines are the Strict DTD

The nexr two lines are the Transitional DTD

And the last two lines are the Frameset DTD

**stevefrm** · 12-21-2009, 05:47 PM

This forum is really really great and I am very very much thankful to you chrishirst .

And I will sure come back to disturb you if i have any doubts.

Again thanks a lot for your help.