SEO for Dynamic Web Sites

Dynamic content itself is not really a problem, because the server does all of the work *before* the page is served to the user agent. However, almost everything that is normally associated with dynamic content and database driven sites is a very real problem. This includes:

• Cookies - Spiders naturally cannot accept cookies
• Forms - including hidden fields, spiders cannot fill out forms.
• Query Strings - Spiders are designed to beware of query strings in URLs

In short, all of the simplest ways of passing values back to the database for generating content are something that the spiders either cannot or will not handle.

Traditionally, the response of promoters to these limitations was to create static pages that the spiders could handle, but obviously, the content is not dynamic, it requires additional work to create static content, and it is very limiting. It is however the simplest of all the solutions.

A better way of handling things is to 'wrap' dynamic content inside a static-seeming page using SSI (Server Side Includes) to bring the dynamic pages inside a static .SHTML file. You can hard-code variables into the SSI calls to bring in limited dynamic content into a form that the spider can handle.

This approach still has the limitations regarding creating additional pages, but at least the content itself is dynamic, and thus updates with the database. You can even hide the nature of the page by creatively adjusting the HTaccess files of the server, making the server treat .HTM files as if they were SHTML, so that you can have a htm extension on the urls, leaving the spider none the wiser about any dynamic aspects to the page.

The ultimate solution however is to make the full dynamic site accessible to spiders, without them ever knowing that its dynamic. This requires the most (and most technical) work, but it means that no special static pages are required at all, and the full dynamic content can be accessed.

The first part of this is to hide any query strings. To do this we must again alter the way that the server handles and parses files, and we must teach the database a new trick. We have to teach the database to place the variables into a new format - one that looks like a standard URL.

One method might be to use a special folder name. For this example lets use 'dynamic'. We would teach the database to encode its variables as if directory names, after the directory called dynamic/ as with the following (non-working) example: The url with query string
http://domain.com/cgi-bin/page.php?databit=value&moredata=value
could become
http://domain.com/dynamic/databit/value/moredata/value/

The disguise is designed for dynamic variables, there would be a script to pull the variables from any URL containing /dynamic/ within its path. Everything that follows the /dynamic/ directory is really just data-pairs seperated by the / character. The htaccess file is used to make the server send this to a script, rather than serve the file.

A simpler method, that is often used in cloaking scripts is to use the htaccess file again to recognise files with a .HTM extension as CGI scripts, while still regarding files with a .HTML extension as standard web pages. In that way, rather than needing to have the 'dynamic' directory you can simply hide variables any way wanted so long as the url ends in a HTM extension, and so long as all genuine HTML files end in a HTML extension.

It is not of course a quick and easy solution. It requires some time and effort to implement, but it works perfectly and enables a lot of things thereafter. The search engines and spiders can happily crawl through as much of the site as they are allowed to, and the URLs are also neater and tidier to the human eye.