How search engines index pages.

The first and most important aspect of search engine optimisation is to understand, at least in a general sense, how the engines work.

A search engine is a very different thing to the web directories. A search engine indexes the whole page in the majority of cases, while a directory simply matches queries against the information in the listing, such as the title, description and url.

When a page is indexed by a search engine, it is not indexed by a living person, but is instead visited by an automatic 'spider' that grabs and assesses the page. Most of the engines have a 'full text' indexing system which means that virtually your entire page is stored in the index.

The amount of time between the submission of a URL to the search engine, and the visit by the spider varies greatly from one engine to another. The time between being visited by a search engine indexing robot, and actually being able to be found in a search on that engine is also something that varies.

In the majority of modern search engines, there is no real advantage to submiting a URL directly at all. The search engines are designed to find documents for themselves by simply following links from known documents. The fastest way to get indexed and ranked is to ensure that you gain some online publicity in the form of links from known websites to yours.

How do search engines index pages?

What can search engine spiders 'see'?

What makes pages rank more highly?

How do I optimise my web pages?

How much optimising will it need?

Is ranking based just on page content?