Review Robots Directives Results in LLMIC
Read restrictions with their sources and collection boundaries.
A robots report is useful when it explains what was observed, where the instruction came from and whether that instruction matches the page’s purpose. A total labelled blocked or non-indexable cannot answer those questions alone. Some restrictions are intentional, some are accidental, and some rows contain no fetched page evidence at all.
Use this review to separate policy decisions from page observations. You will build a small evidence matrix, retain unknown states and choose the next investigation from the actual source of the result. This is an interpretation workflow; it does not prescribe making every URL accessible or removing every noindex instruction.
Identify the Observation You Are Reviewing
Begin with the saved crawl or Robots.txt session that produced the result. Record the collection time, website origin, selected agent and scope. If someone supplies only a screenshot, ask for the underlying source or saved evidence before assigning a fix. A screenshot may omit the tested path, a fetch error or the fact that the source was edited locally.
Distinguish three observation types in your notes: a local robots.txt simulation, a crawler decision that prevented page retrieval, and directives extracted from a fetched response. These observations can relate to the same URL, but they are not interchangeable. A simulation tests source text; a crawl record describes what happened in a specific collection run.
LLMIC’s reviewed crawler creates a blocked record with status code 0 and an explanatory error when robots rules prevent collection. That zero is an application sentinel, not an HTTP status returned by the website. Do not chart it alongside genuine server response codes as though the server sent a new class of HTTP response.
Read the Source Before the Summary
In the Robots.txt workspace, use the source and rule explanation to understand a simulated decision. The summary counts help locate groups and warnings, while the individual agent and path establish the question being tested. Record the exact input text when a decision depends on an exception or a more specific group.
For page observations, inspect the stored Meta Robots and X-Robots-Tag values separately. The former comes from page metadata and the latter represents a response-header field. Keep the raw values even when the interface also provides an indexability label. A label compresses evidence; the original fields are what another reviewer needs to investigate an unexpected result.
When page retrieval was blocked or failed, do not interpret blank directive fields as proof that the page has no indexing instructions. Mark those fields not observed. This small distinction prevents a report from silently turning collection gaps into healthy results, especially when a change to crawl policy alters which URLs can be inspected.
Compare Findings With the Intended Policy
Create a review column for the owner’s intended treatment of each URL group. A deliberately excluded utility page should not receive the same action as an accidentally restricted product page. If the intention is unknown, assign an owner to clarify it before changing the site. Technical detection cannot make that editorial or business decision for you.
| Observed evidence | Review interpretation | Next useful check |
|---|---|---|
| Local robots decision is Blocked | Selected rules restrict this tested input | Confirm agent, path, source and intended policy |
| Crawl row has code 0 and robots error | LLMIC did not fetch page content in that run | Preserve the collection boundary |
| Fetched page contains noindex | An indexing restriction was observed | Confirm it is intentional and inspect its source |
| Successful fetch has no recorded restriction | No restriction was captured in those fields | Review other signals and collection coverage |
| Fetch failed and directive fields are blank | Page directives remain unknown | Resolve retrieval before interpreting absence |
A robots access result and a page’s indexing instruction answer different questions. Google’s documentation distinguishes crawl access from indexing controls. Use that distinction when writing the finding: say which rule or field you observed and avoid collapsing several stages into a claim that the page is definitely absent from, or present in, search results.
Worked Example: Keep Five Unknown Pages Visible
Imagine a deliberately small review of 20 selected URLs. Twelve were fetched without a recorded restriction in the inspected fields, three were fetched with noindex, and five were not inspected because the local crawl respected a matching robots restriction. These are illustrative worksheet categories, not actual LLMIC customer measurements.
The first calculation is coverage: page-level directives were observed for 15 of the 20 selected URLs. Within those 15 fetched pages, three contained noindex. You may report 3 of 15 observed pages, but you must also retain the five unobserved pages. Reporting three affected pages out of 20 as a complete site-wide noindex rate would hide the collection boundary.
A part-to-whole visual can show the three mutually exclusive worksheet categories because every selected URL belongs to exactly one category. It should not combine overlapping counts such as blocked pages, missing titles and noindex pages in one pie chart. Define the categories first; otherwise the visual implies a denominator that the evidence does not support.
Open one representative row from each group. Confirm that its source supports its category, then inspect all three noindex rows against the intended policy. If two are deliberate utility exclusions and one is an accidental public-page restriction, the actionable finding is one confirmed policy mismatch. The raw restriction count and the required-fix count are different measures.
Separate Local Evidence From Search Observations
A fresh LLMIC crawl can verify what the application fetched and stored under its configuration. It cannot, by itself, confirm the search engine’s current index or the identity of a bot that visited your server. If the task concerns Google specifically, use the appropriate Search Console evidence as a separate source and retain its observation time.
Do not reconcile differing tools by selecting whichever number looks better. Compare their inputs: host, URL variants, agent, fetched date, access conditions and whether they used live or cached information. A disagreement can reveal an important timing or scope difference without implying that every source is wrong. Preserve the difference until it has an explanation.
Turn a Result Into a Reviewable Action
For each confirmed mismatch, record the exact URL, intended policy, observed field or rule, evidence time and responsible system. Keep a short action statement such as reviewing the public-page template that emitted noindex. Avoid instructions such as fixing robots everywhere, which conceal the actual change and make accidental collateral effects harder to detect.
Use exporting robots evidence to prepare a handoff. After the owner changes the site, verify the same inputs again. Close the finding only when the originally observed mismatch has been tested with fresh evidence and the intentional restrictions remain understood.
Frequently Asked Questions
Is status code 0 a response from my server?
In the reviewed robots-blocked crawl path, it is a local record indicating that collection was prevented. Read the accompanying error rather than treating zero as an HTTP response code.
Do blank robots fields mean the page is unrestricted?
Only a successfully collected observation can support a statement about the inspected fields. When retrieval failed or was blocked, blank fields should be treated as unobserved.
Should intentional noindex pages count as defects?
Keep them in the observed restriction count, but distinguish them from confirmed policy mismatches. An intentional utility-page exclusion can be a correct configuration.
What denominator should I use for a noindex rate?
State whether the denominator is successfully observed pages or the full selected input set. Also report unobserved URLs so the rate cannot conceal incomplete collection.
Can I use a pie chart for several issue totals?
Only when the categories form a mutually exclusive, complete partition of the stated population. Overlapping issue counts need another presentation, such as separate bars.
Why might the root card differ from my article result?
The reviewed root cards evaluate /. A deeper path can match a different rule, so preserve the exact agent and path when comparing decisions.
Does local indexability prove Google indexed a page?
No. Treat the crawl classification as local evidence about collected signals. Search-engine index observations require their own source and timestamp.
Sources, Statistics and Measurement Notes
The blocked-record behavior was reviewed in CrawlRunner, and page fields were checked in PagesExportService on 5 October 2026. The 20-URL example, its 12/3/5 split and the policy-mismatch scenario are explicitly invented teaching data, not product benchmarks.
Google’s robots meta tag specification supports the distinction between page-level instructions and crawl access. This guide adds an original review worksheet rather than reproducing that specification.