To work around this problem, you can do the following:
Create HTML pages that you can index with the search engines. You can create two types of html pages, "Doorway" pages and "Hallway" pages.
- A Doorway page is a single page that can be indexed by a robot with only one entrance to the site. You can create a doorway page for each product you have. Then you would index those pages.
- Hallway pages (HTML) will be a sitemap of many links to your site. It mentions that Slurp will spider through links so if you add 100 cfm links to your Hallway page all of the links will get indexed. You will want to create one Hallway page for your site that has a collection of links to your entire site.
Once you have created your Doorway and Hallway pages you will then want to submit those pages to the search engines. This will guarantee that your site gets indexed correctly and it gives you more control of what pages get indexed on your site. Slurp should index the site every 60 days so if you maintain the html files it should keep the search engines current.
Tips:
Since Slurp does use links to spider your site you can create invisible links. On the doorway pages create link that match the same color as the background page.
Slurp will not index frames.