Additional Features
Crawler
Updated May 5, 2026
Overview
The Crawler analyzes websites to extract content, structure, and SEO data. Use it to understand your own sites or analyze competitors.
Features
- Website crawling
- Content extraction
- SEO analysis
- Structure analysis
- Document management
- Progress tracking
Accessing Crawler
Navigate to Crawler from the main menu.
Starting a Crawl
- Enter website URL
- Configure crawl settings:
- Depth: How deep to crawl
- Max Pages: Maximum pages
- Respect Robots.txt: Follow robots.txt
- Click Start Crawl
- Monitor progress
Crawl Jobs
Viewing Jobs
- List of crawl jobs
- Status indicators
- Progress bars
- Completion times
Job Status
- Pending: Waiting to start
- Running: Currently crawling
- Completed: Finished
- Failed: Encountered errors
Crawled Data
Viewing Results
- Browse crawled pages
- View page content
- See SEO data
- Analyze structure
Documents
- View all documents
- Search documents
- Filter by type
- Export data
Best Practices
- Start with small crawls
- Respect robots.txt
- Monitor progress
- Review results
- Use for analysis
Related Features
Analyze websites to extract content and SEO data for research and optimization.
Need more help?
Browse our knowledge base for more guides and tutorials