Control Crawl Speed, Scope and Exclusions
Configure LLMIC crawl scope, speed, rendering and exclusions so the audit matches the website you are authorized to test.
A useful crawl is deliberately scoped. More URLs and more speed do not automatically produce better evidence.
Build availability: This guide describes the September 2026 development workflow. Some controls may not yet be included in the public installer. Check the release notes and your app version before following the steps.
Evidence flow
- Define the boundaryChoose the start URL and allowed host scope.
- Set limitsControl page count, depth, speed and redirects.
- Choose renderingUse browser rendering only where it answers a question.
- Record the configurationKeep settings with the resulting crawl.
This visual explains the review sequence. It is not a performance forecast or a set of measured customer results.
Start with a clear question
Open the relevant LLMIC dashboard only after loading the crawl, property or saved observation needed for this review. For website crawler, record the website, date, filters and document layer before interpreting a result. This keeps the work reproducible when another team member reviews it later.
Read the evidence before the recommendation
Open the affected page or row and inspect the retained evidence. Counts help you find patterns, but the page-level source explains what was actually measured. When coverage is partial, a response failed, or a legacy crawl omitted the required field, keep that state visible. Do not turn unavailable evidence into a failed check.
Choose the smallest useful action
Write an instruction that names the exact URL, observed condition and intended result. Preserve intentional behavior and review shared patterns before applying a site-wide change. If a suggestion creates new wording, markup or redirects, a person should approve the facts and destination before publication.
Understand the limits
Crawl only sites you own or are authorized to audit. A capped crawl gives capped coverage, and excluded or failed URLs must not be reported as though they were checked successfully.
Quick reference
| Evidence or state | How to use it |
|---|---|
| Page limit | Set a boundary that fits the audit purpose and plan. |
| Exclusions | Test patterns before running a large crawl. |
| Concurrency | Respect the site's capacity and published policies. |
| Max redirects | Allow enough hops to observe a chain without looping. |
Check the result
After the reviewed change is deployed, collect fresh evidence with a compatible configuration. A task marked complete records workflow progress; it does not prove the live page changed. Save the verification time and result, then use Verify SEO Fixes for supported checks or run a complete follow-up audit.
Continue your workflow
Return to the documentation home, learn how to interpret audit results, or organize reviewed work in the Action Queue and Fix Generator. Remove private information before sharing exports, and contact LLMIC support with a sanitized example when a result cannot be explained.
Frequently asked questions
Configure LLMIC crawl scope, speed, rendering and exclusions so the audit matches the website you are authorized to test.
Load a completed crawl that includes the pages you need to inspect. Confirm the crawl scope, capture time, rendering choice, and any filters before interpreting the result.
Review the affected URL, measured field, source or rendered evidence, and the rule that raised the item. Inspect nearby pages and templates because one shared component can create the same pattern across many URLs.
Create a task with the affected URL, observed evidence, intended outcome, smallest safe change, owner, and verification method. Keep the measured finding separate from the proposed fix.
Publish the approved change, run a fresh compatible crawl, and compare the new evidence with the saved result. A resolved local finding confirms the implementation changed; search performance requires separate observation over time.
The check cannot guarantee rankings, indexing, traffic, conversions, inclusion in an AI answer, or a future search-engine action. Interpret it with site context, search performance, and editorial judgment.
Save or export the evidence, assign the approved action, and use the related guides linked on the page for the next check. After implementation, repeat the same workflow against a fresh crawl so the result is comparable.