OpenAI launched Presence on July 22 around a practical production problem: agent behavior has to keep adapting as products, policies, and customers change. The launch emphasizes evaluations, approvals, escalation rules, and managed changes after an agent goes live.
Two weeks earlier, Mistral introduced version control for prompts and skills, treating them as owned production assets with immutable versions, history, and rollback. The shared signal is clear. A chatbot update needs the same release discipline as any other customer-facing system.
Start every change with this compact release card:
| Field | Required entry | Example |
|---|---|---|
| Release ID | Stable identifier tied to logs and test results | support-bot-2026-07-23.1 |
| Hypothesis | One expected behavior change | Answer the new 2 p.m. shipping cutoff correctly |
| Changed assets | Exact prompts, sources, model, tools, or settings | Shipping page and shipping Q&A pair |
| Excluded assets | What must remain untouched | Refund policy, order lookup, escalation rules |
| Test evidence | Baseline and change-specific results | 24 baseline passes; 8 of 8 cutoff tests pass |
| Rollout | Audience, percentage, channel, and duration | 10% of website chats for 100 relevant conversations |
| Stop condition | Observable reason to halt | One contradictory cutoff answer or failed order lookup |
| Rollback target | Previous known-good release and owner | support-bot-2026-07-18.3, owned by support ops |
If the team cannot fill in one of those fields, the change is not ready to reach customers.
Treat the Whole Chatbot as a Release Unit
A chatbot is more than its system prompt. Its behavior comes from several assets that can change independently:
Instructions define tone, scope, refusal rules, escalation, and how the model should use evidence.
Knowledge sources supply product facts, policies, documentation, and exact answers. A harmless-looking page edit can change retrieval results for unrelated questions.
Models and settings affect reasoning, latency, refusals, tool selection, and response length. A provider update can move several of those at once.
Actions and credentials determine what the chatbot can do outside the conversation. Changing an endpoint, required field, or permission alters the real-world consequence of a reply.
Routing and visibility decide which visitors, domains, languages, or intents see the behavior.
Interface copy shapes consent and expectations before an action, handoff, or data collection step.
Record those assets together as one release manifest, even if they live in different tools. A version that says only “prompt v12” cannot explain a failure caused by a newly crawled policy page or a changed API schema.
The chatbot migration checklist uses a portable manifest to move platforms. The same manifest is useful when the chatbot stays put: it tells reviewers what combination of behavior, knowledge, and access was actually live.
Change One Customer Outcome at a Time
Small changes are easier to test because the team can connect an observed result to a plausible cause. Bundle a new model, rewritten instructions, refreshed sources, and a new action into one release and a regression has four suspects.
Write the hypothesis in customer language:
- “Visitors asking about the cutoff after 2 p.m. receive the next-business-day promise.”
- “Account-specific questions escalate before the bot requests private data.”
- “A cancellation request creates no record until the visitor confirms the exact account and consequence.”
Avoid hypotheses such as “improve the prompt” or “make answers smarter.” They describe activity, not a behavior a reviewer can observe.
Sometimes several assets must change together. A new returns policy may require a page update, an exact Q&A pair, and revised escalation wording. Keep them in one release because they implement one customer outcome. Do not add an unrelated tone cleanup just because the team is already editing the bot.
Match the Release Gate to the Blast Radius
Not every typo needs a committee, and not every settings change is low risk. Classify the effect before choosing the review path.
| Risk class | Typical changes | Minimum evidence | Release gate |
|---|---|---|---|
| Low | Greeting punctuation, suggested-message label, non-policy style example | Preview plus affected-path check | Author can publish; automated record required |
| Medium | Public product source, brand rule, routing condition, model setting | Baseline suite, targeted tests, small canary, named reviewer | Reviewer approves evidence and rollback target |
| High | Pricing, refunds, privacy, account data, tool permission, API action | Baseline and adversarial tests, negative action tests, least-privilege check, monitored canary | Domain owner and technical or security reviewer approve |
| Critical | Payments, destructive actions, regulated advice, broad credential or visibility change | Staging exercise, human confirmation test, incident owner, rehearsed rollback | Two-person approval and explicit production window |
Classify by the worst plausible effect, not by the number of edited characters. Replacing one URL can be critical if it sends customer data to a new system. Rewriting five paragraphs of a public help article may be medium risk if the content is informational and reversible.
Administrative access is a separate concern. The chatbot admin security audit explains how authors, reviewers, publishers, and runtime identities should be separated. Change management uses those roles to move one documented release through its gate.
Build Tests Around the Delta and Its Neighbors
Keep a stable baseline suite for behaviors that every release must preserve: common answers, high-risk policies, missing-source responses, handoffs, and action boundaries. Then add tests for the specific change.
For each new or edited rule, test four directions:
Expected path. Ask the direct question the change is meant to fix.
Boundary. Ask just before and just after a cutoff, eligibility threshold, date, quantity, or permission boundary.
Neighbor. Ask a similar question governed by an unchanged policy. A shipping edit should not alter refunds.
Pressure. Ask the bot to ignore, reinterpret, or make an exception to the new rule.
Save the expected behavior, not one perfect sentence. “State that Friday orders after 2 p.m. process Monday; do not promise Saturday dispatch” survives harmless wording differences. A golden paragraph can fail a good answer simply because the model phrased it differently.
Real failures belong in the suite. The AI chatbot QA guide shows how to turn reviewed conversations into regression cases. Tag each case with the release that introduced it and the later release that fixed it, so the test history explains why the rule exists.
Worked Release: Moving a Shipping Cutoff
Imagine a retailer changes its weekday same-day shipping cutoff from 3 p.m. to 2 p.m. Support updates the shipping page, but an old Q&A pair still promises same-day handling until 3 p.m.
The release owner writes:
release: support-bot-2026-07-23.1 hypothesis: > Orders placed after 2 p.m. are described as processing on the next business day, including Friday orders that move to Monday. changed: - source: /help/shipping-times - qa: shipping-cutoff excluded: - refund-policy - order-status-action tests: baseline: 24/24 pass targeted: 8/8 pass canary: traffic: 10% review_after: 100 shipping conversations stop: - any answer promises same-day handling after 2 p.m. - order-status action failure rate exceeds the current baseline rollback: support-bot-2026-07-18.3 owner: support-operations
One targeted transcript makes the boundary concrete:
Customer: If I order at 2:30 p.m. this Friday, will it ship today? Expected behavior: - Say the order is after the 2 p.m. cutoff. - State that processing starts the next business day, Monday. - Do not promise a delivery date without destination and service level. Observed: "It is past today's 2 p.m. cutoff, so processing starts Monday. Delivery timing depends on your destination and selected service." Result: pass
The team also asks about a 1:59 p.m. Friday order, a 2:01 p.m. Monday order, weekend orders, expedited delivery, and refunds. That last neighbor test matters because the release should not disturb a policy it did not intend to change.
During the canary, reviewers filter for shipping conversations rather than judging all traffic. After 100 relevant conversations, suppose 96 answers pass, three correctly escalate because the destination is missing, and one still quotes the old 3 p.m. rule from a cached PDF. The stop condition fires. The team rolls back, removes or replaces the stale source, reruns the eight targeted tests, and starts a new release ID. The failed release remains in history; it is not edited into appearing successful.
Canary the Risky Path, Not Just Random Traffic
A random 5% rollout can miss a rare but important intent. If only one in 200 visitors asks about refunds, a small general canary may produce no evidence about the refund change.
Route the canary around the affected path when the platform allows it: a specific intent, language, action, source collection, authenticated audience, or channel. If targeted routing is unavailable, keep the change in a playground or staging chatbot and recruit internal reviewers to exercise the exact path before a small general rollout.
During the canary, compare against the previous release:
- answer pass rate on the affected intent;
- escalation rate and whether the handoff contains useful context;
- action attempts, confirmations, failures, and duplicate effects;
- response time and conversation length;
- negative feedback, repeated questions, and abandonment near the changed step.
Set thresholds before publishing. “Watch feedback” invites debate after something goes wrong. “Stop after one unauthorized action, one critical policy contradiction, or a five-point drop in the targeted pass rate” gives the on-call owner a decision they can execute.
Preserve the Evidence Behind Every Version
The version history should answer six questions without reconstructing the change from chat messages:
- What customer behavior was supposed to change?
- Which assets changed together?
- Who authored, reviewed, and published them?
- Which tests passed, with which inputs and expected behaviors?
- Which production conversations ran on the version?
- What caused promotion, rollback, or replacement?
Keep immutable snapshots of the release manifest and test results. Store links to representative conversations, but follow your retention and access rules rather than copying private customer data into a broad project document.
When a source refresh happens automatically, create a release event for it. Record the crawl time, changed pages, parse failures, deleted content, and evaluation result. “No one edited the prompt” does not mean the chatbot stayed the same.
The rollback target must also be deployable. A label pointing to a deleted model, expired credential, or unavailable source snapshot is documentation, not recovery. The model rollback checklist covers how to keep a tested fallback ready when the model itself is the moving part.
Make Each Release Easier to Explain
Good chatbot change management creates a chain from customer evidence to a narrow hypothesis, from that hypothesis to a tested release, and from the release to observable production results. It lets teams improve quickly because every update has an owner, a boundary, and a safe way back.
In Agentkit, conversation logs and analytics provide evidence for the review loop, while Q&A pairs let teams pin corrected answers when a policy requires exact wording.
No credit card required.
