 |
|
|
|
|
|
|
|
|
You can specify exactly which pages to spider, as well as
which ones not to. You can specify which pages to include with your Primary URL and Secondary URLs
settings. Just as easily, you can list pages that you don't want to be included within Exclusion
URLs/Masks.
You can also determine whether or not you want the siteLevel spider to obey your site's robots.txt file.
With siteLevel, you can also spider password protected areas of your site. You simply need to
provide a username and password for the given area(s) within the Password Authenication setting of your
account management area.
NOTE: If you do choose to spider a password protected area of your site, visitors to your site will still
be required to have a valid username and password to enter the protected area. If a search result
for a protected area comes up from your site search box and a user clicks on the result, they will be
prompted, as before, for their username and password. |
|
|
|
|
|
|
|
|
|
