How to Run a Website SEO Audit With the OutreachBox Crawler

Quick answer: Open the Crawler, enter the website URL, set the crawl depth, max pages, and whether to respect robots.txt, then click Start Crawl. When it finishes, review the crawled pages, content, structure, and SEO data to find thin content, missing metadata, and structural issues to fix.

The crawler analyzes websites — yours or competitors' — to extract content, structure, and SEO data. It's the fastest way to audit a site and build a content inventory inside OutreachBox.

How to start a crawl

1. Navigate to the Crawler (or enable Auto Start Crawl when creating a project).
2. Enter the website URL.
3. Configure the crawl:

• Depth — how many link levels deep to follow.
• Max pages — a cap to keep crawls fast and focused.
• Respect robots.txt — follow the site's crawl rules.

1. Click Start Crawl.
2. Monitor progress as the job runs.

Understanding crawl jobs

Each crawl is a job with a status:

How to review crawl results

When a crawl completes:

1. Browse the crawled pages and open any page to view its content.
2. Review the extracted SEO data and page structure.
3. Use Documents to search, filter by type, and export the data.
4. In a project, find this under the Crawled Data tab.

What to look for in your audit

• Thin or missing content on important pages.
• Structural issues like orphaned pages or shallow hierarchy.
• Metadata gaps to fix for better search visibility.
• Content inventory to plan new or updated pages.

Best practices

• Start with a small crawl (low depth, capped pages) to validate settings before a full crawl.
• Respect robots.txt, especially when crawling sites you don't own.
• Re-crawl periodically to catch new issues as the site changes.
• Pair crawl findings with keyword research to prioritize fixes.

Frequently Asked Questions

Why did my crawl fail?

Common causes are an inaccessible or invalid URL, the site blocking crawlers, or the crawler service being temporarily unavailable. Verify the URL loads in a browser and try again.

Can I crawl a competitor's website?

Yes — the crawler works on any reachable site. Respect robots.txt and use the data for analysis. See How to Do Competitor Analysis for SEO.

How many pages can I crawl?

Set the Max pages limit per crawl. Start small to validate, then increase for a full inventory.

Related articles

• How to Create an SEO Project
• How to Do Competitor Analysis for SEO
• How to Do SEO With OutreachBox

---

Crawl first, then fix — an audit turns guesswork into a prioritized list of SEO improvements.