Quick answer: Open the Crawler, enter the website URL, set the crawl depth, max pages, and whether to respect
robots.txt, then click Start Crawl. When it finishes, review the crawled pages, content, structure, and SEO data to find thin content, missing metadata, and structural issues to fix.
The crawler analyzes websites — yours or competitors' — to extract content, structure, and SEO data. It's the fastest way to audit a site and build a content inventory inside OutreachBox.
Before you start
Use a website you are authorized to audit and confirm the URL is publicly reachable. Start with a limited scope when reviewing a large site.
Practical example
Fix a broken internal link on a valuable service page before adjusting minor wording across hundreds of posts. Record the affected URL, the change, and a follow-up crawl result.
Check your result
If the crawler returns few pages, inspect redirects, authentication, robots rules, and navigation links. A successful crawl does not mean Google has indexed the same pages.
How to start a crawl
- Navigate to the Crawler (or enable Auto Start Crawl when creating a project).
- Enter the website URL.
- Configure the crawl:
- Depth — how many link levels deep to follow.
- Max pages — a cap to keep crawls fast and focused.
- Respect robots.txt — follow the site's crawl rules.
- Click Start Crawl.
- Monitor progress as the job runs.
Understanding crawl jobs
Each crawl is a job with a status:
| Status | Meaning |
|---|---|
| Pending | Waiting to start |
| Running | Currently crawling |
| Completed | Finished — results are ready |
| Failed | Encountered errors |
How to review crawl results
When a crawl completes:
- Browse the crawled pages and open any page to view its content.
- Review the extracted SEO data and page structure.
- Use Documents to search, filter by type, and export the data.
- In a project, find this under the Crawled Data tab.
What to look for in your audit
- Thin or missing content on important pages.
- Structural issues like orphaned pages or shallow hierarchy.
- Metadata gaps to fix for better search visibility.
- Content inventory to plan new or updated pages.
Best practices
- Start with a small crawl (low depth, capped pages) to validate settings before a full crawl.
- Respect
robots.txt, especially when crawling sites you don't own. - Re-crawl periodically to catch new issues as the site changes.
- Pair crawl findings with keyword research to prioritize fixes.
Frequently Asked Questions
Why did my crawl fail?
Common causes are an inaccessible or invalid URL, the site blocking crawlers, or the crawler service being temporarily unavailable. Verify the URL loads in a browser and try again.
Can I crawl a competitor's website?
Yes — the crawler works on any reachable site. Respect robots.txt and use the data for analysis. See How to Do Competitor Analysis for SEO.
How many pages can I crawl?
Set the Max pages limit per crawl. Start small to validate, then increase for a full inventory.
Sources
- Google Search Central: Creating helpful, reliable, people-first content
- Google Search Central: AI features and your website
Related articles
Crawl first, then fix — an audit turns guesswork into a prioritized list of SEO improvements.