Over the last little while many people have asked me to make search engines in PHP/MySQL for them. From what I know PHP/MySQL isn't really a good language to search in but it did get me thinking. How do extremely efficient, fast search engines such as Google work under the hood? I mean almost every site in the world is on there. How does it return a result that fast?
The only plausible solution I have though of is pre-grab results. At first I thought this was a probable solution but you type in something like 'adfkjddikwiodjnfkdk' and still get a fast result. It's possible but I'm not sure. And even then it would take forever to fetch such a large result source from a database.
I'm not sure why but this is really bugging me. Any ideas?

I believe google use mostly python. But with the amount of hardware they have efficientcy in thier code isn't that important.

I think they have highly efficient code, but also hundreds of thousands of computers come servers, and great load balancing.

Also Google does not use a relational database model with table structures as you would be doing with MySql.

Their DB model is the inverted index which is far quicker on data retrieval but slower to update and index.

I've always marveled at how fast Google can return results, too! Although I'm not well-versed in the working of SEO's, so I just marvel at it in awe! lol.

I have always wondered though why a forum of mine (PHP) just won't get indexed. It is a local forum (city issues, etc.) but still, you'd think it would at least get indexed, would you???

Originally Posted by fitnfree

I've always marveled at how fast Google can return results, too! Although I'm not well-versed in the working of SEO's, so I just marvel at it in awe! lol.

Millions of little tiny gremlins...

Matt you really answered your own question. They do pre-fetch things. That's what indexing is all about. When you go to Google and type in a query, Google doesn't go out and search the web. They search their internal servers.

I think that's the main thing you're looking for if I'm understanding right.

Naturally they want to return results as fast as possible. I'm sure they keep working to make their code more efficient and do what they can to remove bottlenecks. Like Chris said they search an inverted index database model.

So they do everything they can to make the search of they're index faster, but they are searching their index and not going out on the web finding results in real time. When you search you are searching the web as Google saw it sometime prior to your search. For some sites that might mean a few minutes ago. For others it might mean a few months ago.

Millions of little tiny gremlins

Naah, it's not the gremlins, that's MSN

It's the Pigeons

Sweet link...

Although I'm having to bite my tongue to refrain from the obligatory comment on how Google search is completely "bird brained."