Configure Crawl Speed in LLMIC
Choose a sustainable speed using useful completions and server health.
A crawl that sends more requests is not necessarily a crawl that finishes more useful work. If the website starts returning errors, or the Mac spends longer rendering complex pages, the number of usable observations can flatten while request activity keeps rising.
Configure crawl speed in LLMIC around successful work and server health. Start with a bounded sample, record the current settings, and adjust one relevant control at a time. Your aim is a sustainable investigation that produces the evidence you need, not the highest number displayed during a short burst.
Choose a Representative Test Before Changing Speed
Use a known URL List when you want to compare configurations. Include the page types that matter to the audit: a simple article, a category, a product or another representative template. A list containing only lightweight pages can make a configuration look suitable for a heavier workload than you actually intend to run.
Keep the list fixed during the comparison. Record the URL count, rendering choices and any extra checks. If one run follows additional links while another inspects a fixed list, their completion rates do not describe equivalent work.
Agree an appropriate test window with the website owner when the site is sensitive to load. Avoid treating a production crawl as an unlimited stress test. The configuration should fit the authorized investigation and the site’s capacity.
Find the Controls That Affect Request Pressure
Open More, then Settings, then Crawling. In Performance & Storage, expand Advanced crawler controls. The configuration includes Concurrent requests, Per-host delay, Respect robots Crawl-delay and Adaptive concurrency (auto speed). Check the values in your installed version before beginning.
| Control | What it changes | What it does not guarantee |
|---|---|---|
| Concurrent requests | The configured upper bound on simultaneous request work | A fixed number of completed pages each second |
| Per-host delay | Spacing imposed on requests to a host | That a slow response becomes faster |
| Adaptive concurrency | Automatic adjustment within the configured maximum | Unlimited capacity or freedom from errors |
| Rendering and extra checks | The work required for the chosen investigation | An equivalent workload when disabled |
A concurrency value is not a throughput measurement. Responses take time, some work waits, and page processing can continue after a response arrives. Do not turn a concurrency setting into a promise about how many pages a customer will finish per minute.
Keep Politeness and Access Rules Separate
Respect robots.txt and Respect robots Crawl-delay are different settings. The former concerns crawl restrictions; the latter concerns a delay requested through that directive. Do not switch either off merely to make a speed comparison look better.
For an authorized website, agree the request behavior with its owner and document any configuration change. Different crawlers interpret delay conventions differently. This guide describes the LLMIC settings and does not imply that Googlebot follows the same controls.
If the server returns HTTP 429, investigate the rate limit and any Retry-After information. Increasing concurrency is not the appropriate automatic response to a server asking for less traffic. Keep request failures visible in the comparison instead of counting every attempt as progress.
Preserve the Checks Needed for the Actual Question
Rendering and external enrichment can add work. Turning them off can make a run shorter while also removing the evidence the audit needs. Decide which checks are necessary before tuning speed, then keep them consistent across candidate settings.
When only a few pages require a heavier investigation, consider a separate focused run for those pages. Record the separation clearly. A fast HTML-oriented inventory and a rendered diagnostic sample answer different questions and should not be presented as interchangeable results.
Watch the Mac’s workload as well as the website’s responses. Memory pressure, other applications and local processing can affect elapsed time. A server may be healthy while the client machine is busy, so request settings alone may not explain a slow completion rate.
Worked Example: Find the Useful Plateau
This chart is an illustrative experiment, not an LLMIC benchmark. Four candidate settings use concurrency values of two, four, six and eight. The hypothetical runs attempt 20, 38, 52 and 60 requests per minute, while useful completed observations rise from 18 to 34, then flatten at 35 and fall to 30.
The separation matters more than the highest attempted-request value. In this constructed example, raising concurrency beyond four produces little useful gain and then a decline. The next step is to inspect errors, waiting and processing conditions rather than choosing eight because it generated the most activity.
The difference between attempts and useful completions is not automatically an error count. Some work may still be in progress, some attempts may be retries, and the two measures need compatible definitions. Preserve the underlying records before assigning a cause to the gap.
Use the chart as a way to design your own comparison. Your website, Mac, page mix and settings will produce different results. No concurrency value in the illustration is a universal recommendation.
Save the Chosen Configuration With Its Reason
Record the selected setting, the tested URL set, elapsed time, usable outcomes and observed failures. Explain why the setting was chosen. “More useful completions with no observed increase in failures on this sample” is a bounded claim that another operator can review.
Also record when to reconsider it. A new rendering requirement, a slower server, a different section or a larger page mix may invalidate the earlier comparison. A configuration is a working choice for defined conditions, not a permanent certificate of safety.
Continue with Review Crawl Speed Results in LLMIC to distinguish transient activity from sustained useful progress. For the URL boundary itself, use Configure Crawl Scope in LLMIC.
Questions About Speed Settings
Is more concurrency always faster?
No. It can increase contention or server pressure, and useful completions may stop improving. Compare a fixed workload and inspect failures rather than relying on the configured maximum alone.
What does Per-host delay help control?
It spaces request activity to a host. It does not remove server processing time or local analysis work. Record it alongside concurrency so the comparison can be repeated.
Should I disable rendering to improve the result?
Only if rendering is unnecessary for the question being tested. Otherwise the faster run may be performing a different, less complete investigation. Keep required evidence collection consistent.
Does adaptive concurrency override my maximum?
The implementation adjusts within the configured concurrent-request ceiling. Treat that ceiling as a limit rather than a target throughput, and inspect the actual outcome of the run.
What should I do when 429 responses appear?
Reduce pressure and review the site’s rate-limit response, including Retry-After when supplied. Coordinate with the owner instead of repeatedly increasing request activity.
Can another application on my Mac affect the comparison?
Yes. Local CPU, memory and network use can alter completion time. Record relevant conditions and repeat a suspicious result before attributing it to the website or one setting.
Which setting should I keep after the test?
Choose the configuration supported by useful completions, acceptable failure behavior and the required audit scope. Preserve the evidence and avoid choosing solely by peak speed.
Sources, Statistics and Measurement Notes
Control names and adaptive-concurrency boundaries were checked against LLMIC’s crawling-settings implementation on 5 October 2026. Chart values are constructed teaching data, not measured product performance or customer results.
MDN’s HTTP 429 reference, accessed 5 October 2026, explains rate limiting and Retry-After. The illustrative comparison does not attribute its completion gap to a specific HTTP status.