Crawl website content and add it to Knowledge Base in Acrobat collections.
The Web Indexer capability is provided through the Knowledge Base Indexer desktop application. Use it to test a crawl without uploading content or to crawl intranet and internet websites and upload the indexed pages to a collection. The application can also crawl authenticated websites.
How it works
The Knowledge Base Indexer starts from one or more HTTPS URLs and follows links according to the crawl settings you define.
You can control:
- The starting URLs.
- How deeply the indexer follows links.
- The maximum number of pages it processes.
- How many pages it processes in parallel.
- Whether it authenticates before crawling a protected website.
- Whether it discovers content or uploads it to a collection.
The logs show crawl progress, discovered and processed pages, warnings, errors, and completion status.
Choose an operation mode
Discovery Mode
Discovery Mode crawls and analyzes website content without uploading it.
Use this mode to:
- Confirm that the indexer can access the website.
- Monitor which pages are discovered and processed.
- Identify invalid URLs, blocked pages, warnings, or other crawl problems.
- Test the crawl scope before uploading content.
Discovery Mode is selected by default.
Upload Mode
Upload Mode crawls website content and uploads the indexed data to a selected collection.
To use Upload Mode, sign in to Knowledge Base in Acrobat and select a collection where you have write access. The application opens a browser for authentication and lets you refresh the list of available collections when needed.
Supported systems
The Knowledge Base Indexer is available for:
- Intel-based Mac systems.
- Apple Silicon Mac systems.
- Windows systems.
See Install the Knowledge Base Indexer to choose and download the correct package.
Configure a crawl
Enter one or more HTTPS URLs, with each URL on a separate line. Then configure how the Knowledge Base Indexer processes the website:
- Crawl Depth: Controls how many levels of links the indexer follows from the target URLs. A value of 1 crawls pages linked directly from the starting URLs. Higher values follow additional levels of links.
- Page Limit: Sets the maximum number of pages processed during the crawl. The default is 5, and the maximum is 5,000.
- Number of Workers: Sets how many crawlers run in parallel. The default is 4. Use 4–8 for most systems, depending on the available processor cores.
- Authentication URL: Provides a URL the indexer can use to authenticate before crawling protected content. Public websites do not require this value.
Higher crawl depths and page limits can greatly expand the crawl. Increasing the number of workers can reduce processing time but uses more system resources.
Only HTTPS URLs are supported. The indexer rejects local addresses, loopback addresses, and certain restricted IP addresses.
Use indexed website content
Upload Mode adds indexed website content directly to a collection. After processing finishes, users with access to the collection can use that content with supported search and AI Assistant features.
The Web Indexer differs from adding a Public Website source. A Public Website source retrieves search-engine content when a query runs and does not crawl or store an entire website in advance.
Best practices
- Test the crawl in Discovery Mode before uploading content.
- Begin with a shallow crawl and a small page limit.
- Target specific website sections instead of entire domains.
- Review the logs for errors, warnings, and unexpected crawl behavior.
Things to know
- An internet connection is required.
- Upload Mode requires a Knowledge Base in Acrobat account.
- You can upload only to collections where you have write access.
- Deep crawls, large page limits, and high worker counts can increase processing time and resource use.
- If authentication expires during processing, the application attempts to refresh the session.
