Mohith G's Freshness in RAG: keeping the index in sync with the world explains how to keep an index current, including removing documents that no longer belong. This guide picks up at the deletion decision. What evidence should you require before a URL missing from a crawl becomes a removal from your index?
Treat a missing URL as a review item. Confirm the source's status, the crawl's coverage and the intended access rules before removing indexed content. A snapshot comparison can identify what needs attention, but cannot authorize deletion by itself.
Separate absence from deletion
A crawler can omit a page because it hit a page limit, lost access or followed a different set of links. It can also report a failed request for a page that still exists. Neither observation proves that the source owner removed the document.
Use this decision table as a review policy for web-source exports. It is a suggested procedure, not a measured customer outcome or an automatic Agentkit indexing feature.
| Evidence | Next check | Index decision |
|---|---|---|
| URL is absent from the latest export | Check crawl completion, page limits, filters and permissions; fetch the source again | Hold the deletion decision until the absence is explained |
| Fetch failed or returned empty text | Inspect the response and extractor; retry with the intended access | Do not replace useful content with a failure page |
| Source owner confirms removal, or the authoritative source reports a deliberate deletion | Check source identity, replacement location and affected chunks | Remove or exclude the confirmed source, then verify retrieval |
| Source moved to a new URL | Confirm the destination and its content with the source owner or redirect | Replace the source mapping and retire obsolete chunks |
| Access was revoked | Verify who may still retrieve the indexed content | Enforce the access change immediately; do not wait for a crawl retry |
| Text changed successfully | Inspect whether the change is substantive and extraction is complete | Update the affected content, then test answers against the new version |
Holding a deletion decision does not mean serving old information indefinitely. If a policy or price cannot be verified within your freshness requirement, stop relying on it for answers and route the question for review. Revoked access needs immediate enforcement even if the source remains online.
Compare the exports before changing the index
Save the export used for the current index and the proposed replacement. Record their crawl scope, extraction settings and completion status. Differences in those settings can make the comparison misleading.
Open the RAG snapshot checker and load both JSON or JSONL files. For successful records, supply a URL and exactly one text field named content, markdown or text. An optional numeric status records the HTTP response. For example, a successful record could look like this:
{"url":"https://example.com/help/returns","content":"Returns are accepted within 30 days.","status":200}
This example is illustrative. Use your actual source exports for a review. Keep failed fetches in the export when your crawler provides them, rather than silently dropping those records.
The checker groups records as added, changed, unchanged, missing or failed. It compares exact extracted text, so a changed footer can produce a difference. Unchanged text does not establish that a policy is still valid. The tool runs the comparison in your browser without uploading the source content and can download a review report. It does not query your source system, inspect your vector database or delete anything.
Investigate each missing or failed record using the table above. A report with no differences is still only evidence about the supplied exports, not proof that your live index matches them.
Keep a decision record for each removal
Beside the comparison report, record the source URL, the observation, the confirming evidence and the intended action. Name the reviewer and record the time. For a moved page, include the replacement URL. For an access change, record the applicable permission decision without putting private content into a shared report.
A useful record answers one question: why did this source stop being eligible for retrieval? “It disappeared from the export” leaves the next operator guessing. “The source owner confirmed retirement and identified the replacement” gives them something they can verify.
Verify the result in retrieval
After applying the approved index change, search for a distinctive passage from the retired source and check the returned source IDs. Then ask a question that previously depended on it. Confirm that the answer uses the replacement, acknowledges the missing information or follows your escalation policy.
Test permissions separately with the relevant user roles when access changed. A correct answer from an administrator does not establish that restricted content is hidden from other users. Keep the previous index version only under the same access and retention rules, so rollback does not restore revoked access.
For the earlier steps, see our guide to checking source documents before indexing. For this decision, start with the snapshot checker, investigate the flagged sources and save the evidence before applying removals.
