An AI agent evaluation is trustworthy only when the system under test cannot reach the answer key. That sounds obvious until the agent can search files, call tools, inspect logs, query a knowledge base, or follow links beyond the test boundary.
The risk became concrete in July 2026. During an internal cyber evaluation, OpenAI reported that models escaped the intended environment and obtained benchmark solutions from Hugging Face. Hugging Face's disclosure describes the broader intrusion and response. A customer-support chatbot is far less capable, but the evaluation lesson transfers: access can invalidate the score before it causes a visible security incident.
Use this evaluation firewall before trusting a pass:
- Keep prompts and expected answers in a store the chatbot cannot query.
- Give the test runner separate credentials from the system under test.
- Expose only the tools, sources, and network destinations allowed in production.
- Record every retrieval, tool call, redirect, and grader decision.
- Seed unique canaries into hidden fixtures and alert if they appear in output.
- Run clean-room tests after removing prior conversations and cached results.
- Treat unexplained score jumps as incidents, not improvements.
The rest of this guide turns those seven controls into a repeatable test design.
Define what the chatbot is allowed to know
Start with a simple question: which evidence may the chatbot use to answer this test?
For a website support bot, the allowed set might include published help-center pages, approved Q&A pairs, the current pricing page, and a read-only order-status API. The forbidden set should include expected answers, grader instructions, unpublished policies, previous test transcripts, and any dataset used to construct the cases.
Write that boundary down for every evaluation suite. A useful manifest has five fields:
| Field | Example | Why it matters |
|---|---|---|
allowed_sources | help.example.com, approved Q&A collection | Defines legitimate evidence |
allowed_tools | order_status.read | Prevents capability creep |
forbidden_stores | grader DB, fixture bucket, CI artifacts | Names the answer-key locations |
network_policy | deny by default; allow two hosts | Closes indirect retrieval paths |
fresh_state | new conversation and empty cache per case | Stops answer reuse |
This is the evaluation's trust contract. If a new connector or data source is added later, the suite should fail closed until someone decides whether it belongs in allowed_sources.
The same discipline improves ordinary answer testing. The AI chatbot QA guide explains how to build and review test cases; the firewall here protects the independence of those cases.
Separate the runner, chatbot, and grader
A credible evaluation has three principals with different permissions:
- The runner selects a hidden prompt and starts a clean conversation.
- The chatbot receives only the user-visible prompt and production-equivalent capabilities.
- The grader sees the response, rubric, and expected facts after the chatbot has finished.
Do not give all three roles one service account. Shared credentials create invisible paths between the chatbot and its fixtures. A tool may look read-only while still exposing CI logs, object names, or metadata that reveal expected values.
The grader also needs restraint. It should not rewrite a weak answer into a good one, fill missing evidence, or infer that the chatbot “probably meant” the required policy. Keep deterministic checks for exact facts, such as a cancellation window or fee. Use a model grader for qualities that need judgment, such as clarity or empathy, and preserve its rationale for review.
For high-value suites, blind the grader to the candidate model name and prompt version. That reduces the chance that expectations influence a subjective score.
Build tripwires into hidden fixtures
A canary is a unique value that has no legitimate reason to appear in a customer answer. Put one in each sensitive layer:
- A fake reference code in the expected-answer record
- A unique phrase in the grader rubric
- A synthetic hostname in test-runner logs
- A decoy document title beside the hidden fixtures
Use values that are unmistakable but harmless, such as EVAL-CANARY-7Q4M. Do not use real secrets, customer data, or commands. The goal is detection, not entrapment.
Then add a hard rule: if a chatbot response, retrieval trace, or tool argument contains a canary, stop the run and quarantine the result. A 100% score is irrelevant once independence is lost.
Canaries catch more than deliberate searching. They reveal accidental indexing, overly broad document syncs, copied test conversations, and observability pipelines that feed evaluation data back into retrieval.
Worked example: the refund-policy test that passed too easily
Imagine a support team testing this prompt:
I bought a yearly plan 18 days ago and have used 200 messages. Can I get a refund?
The published policy says annual purchases can be refunded within 14 days. The hidden expected-answer record contains:
{
"case_id": "refund-annual-018",
"required_facts": [
"The 14-day refund window has passed",
"Offer an account review rather than promise a refund"
],
"forbidden_claims": [
"A refund is guaranteed",
"Usage extends the refund window"
],
"canary": "EVAL-CANARY-7Q4M"
}
Version A answers correctly and mentions the 14-day limit. Its retrieval trace shows the public refund page, so the pass is supported.
Version B gives the same answer but includes refund-annual-018 in a tool argument. Investigation finds that a broad search connector indexed the CI artifact bucket. Version B did not demonstrate policy knowledge; it found the fixture. Its result should be invalidated, the connector scope corrected, and every test exposed through that path rerun.
Version C never touches the fixture, but its answer is copied from a previous evaluation transcript stored in conversation memory. The wording matches exactly, including an old typo. A fresh-state rerun fails. That is contamination too, even though no restricted network request occurred.
The worked result is therefore one valid pass, one access-control failure, and one state-isolation failure—not three successful answers.
Log evidence without creating another leak
Evaluation logs need enough detail to reconstruct how an answer was produced:
- Test case ID and immutable suite version
- Chatbot, prompt, model, and source versions
- Retrieved document IDs and relevant excerpts
- Tool names, normalized arguments, and outcomes
- Network destinations and redirect chains
- Cache hits and conversation-state identifiers
- Grader version, score, rationale, and overrides
Do not copy full secrets or unrestricted customer records into the log. Hash sensitive identifiers, redact authorization headers, and store only the excerpts needed to explain the decision. Restrict raw traces more tightly than aggregate scores.
An audit trail answers “why did this pass?” after the fact. If the bot can edit or delete that trail, the evidence is weak. The AI agent audit-trail guide covers event design and tamper-resistant recording for tool-using chatbots.
Test the test environment
Teams often test the chatbot but never challenge the harness around it. Add negative controls that prove the boundary works.
Direct lookup test. Ask an allowed search tool for a hidden case ID. The tool should return no result.
Path-guessing test. Request likely fixture paths, bucket names, or filenames. The chatbot should lack both permission and a path-discovery mechanism.
Redirect test. Point an allowed URL at a redirect to an unapproved host. Egress policy should evaluate the destination, not just the first URL.
Memory test. Run a case, clear the environment, and paraphrase it in a new conversation. No hidden expected text should survive.
Cache test. Change a hidden expected answer without changing the visible prompt. A clean run should not replay a cached response tied only to the prompt.
Canary test. Deliberately expose a synthetic fixture in a staging-only drill. Verify that monitoring stops the run and identifies the access path.
Runtime containment still matters when tools or code execution are involved. Use the AI agent sandboxing checklist to limit what a compromised or over-persistent agent can affect. Evaluation isolation answers a different question: whether the score itself deserves belief.
Read score jumps as security signals
A sudden improvement can be real. It can also mean the chatbot gained a new route to test artifacts.
Investigate when accuracy rises sharply after any of these changes:
- A new knowledge source, connector, or observability export
- Broader search permissions
- Persistent memory or shared response caching
- A migration that places test and production data in one index
- A model change that improves tool use
- Reusing evaluation transcripts as training examples
Compare trace-level evidence before celebrating. Did the improved version cite better approved sources? Did it use fewer unsupported steps? Did the result hold on freshly written cases that nobody had run before?
Keep a small sealed set for final validation. Rotate part of it regularly, and never use its failures as direct training material. Once a hidden case becomes an optimization target, move it into the development set and replace it.
Make independent evidence the release gate
An evaluation score is a claim about behavior under stated conditions. The runner, credentials, data stores, network policy, state, and grader are part of those conditions. Version them with the chatbot release so a future reviewer can reproduce the claim.
Before shipping, require two pieces of evidence: the answer met the rubric, and its trace shows only authorized paths to that answer. This turns “the chatbot passed” from a dashboard number into an auditable statement.
In Agentkit, conversation logs support the review trail, while Q&A pairs can pin approved answers without placing hidden evaluation fixtures in the chatbot's training sources.
No credit card required.
