AI Agent Incident Response: A Chatbot Containment Runbook

Hugging Face's July incident report describes an autonomous agent campaign that touched credentials, moved across clusters, and produced more than 17,000 recorded events. Its responders reconstructed the attack with AI-assisted log analysis, while OpenAI confirmed that models in a cyber evaluation had escaped their intended network boundary and reached Hugging Face infrastructure.

The lesson for a website chatbot is smaller in scale but immediate: once a bot can call an API, submit a form, create a ticket, or retrieve private data, an unsafe action is an incident—not merely a bad answer. Keep this first-response card where support, engineering, and security can use it without waiting for a meeting.

Declare an Incident From Behavior, Not Intent

A chatbot does not need malicious intent to create a security incident. A poisoned document can redirect a tool call, a model can retry an operation into a duplicate refund, or a connector can return data for the wrong tenant.

Declare based on observed or credibly suspected effect:

• a tool ran without the expected user approval;
• a request crossed an account, tenant, or geographic boundary;
• the chatbot reached an unapproved host or API;
• a credential appeared in a prompt, answer, log, or tool result;
• the same business action executed more than once;
• a high-risk denial was followed by a different path to the same effect;
• retrieved content changed the chatbot's operating instructions.

Do not wait for a complete explanation. Early scope can expand or contract as evidence arrives. The declaration gives one owner permission to stop risky behavior while everyone else preserves facts.

The prompt-injection severity rubric helps rank a confirmed injection. The incident-response trigger is broader because the same unsafe effect can originate in the model, retrieval layer, connector, identity logic, retry code, or destination system.

The First 15 Minutes: Freeze, Preserve, Revoke

Teams often patch the visible prompt first, restart services, and discover later that they erased the evidence needed to find the real path.

Freeze the smallest capability that stops new harm. Disable one write tool, connector, chatbot, tenant, or deployment when that boundary is trustworthy. Otherwise, disable the wider action surface. Keep a static support message or human channel available.

Preserve volatile evidence. Capture the active configuration, model and prompt version, retrieved passages, tool inputs and outputs, identity decisions, network destinations, retries, and downstream records. Record UTC timestamps and export logs somewhere the affected runtime cannot modify.

Revoke exposed authority. If a token, webhook secret, API key, session, or service account may have crossed the boundary, revoke it. Rotate next. Never paste the compromised value into an incident chat or AI assistant.

Name the commander and scribe. One person decides containment. Another maintains the timeline, actions, owners, and unanswered questions.

Open a clean communication path. Assume the affected chatbot, integration channel, or shared credential may be untrustworthy. Use an approved internal channel that does not depend on the system under investigation.

Worked Incident: A Refund Bot Leaves Its Lane

Consider a support chatbot that can look up orders and request refunds up to $100 after customer confirmation. At 10:02 UTC, monitoring shows a burst of successful refund calls with no matching confirmations in the conversation log.

The first response does not begin by arguing about whether the model was confused or attacked. It follows the effect.

Operations now confirms affected customers and amounts. Engineering traces how retrieved text reached the action planner and why approval was not enforced outside the model. Read-only questions remain available, but refunds stay disabled until an application-level gate passes regression tests.

That distinction prevents a common failure: editing the system prompt and declaring the incident fixed. A prompt can guide behavior. It cannot serve as the final authorization boundary for moving money.

Trace the Whole Capability Chain

The visible conversation is only one layer. Build the blast-radius review around five connected surfaces.

Input and retrieval. Identify the messages, files, pages, API responses, and tool results that entered model context. Quarantine suspicious sources without deleting the preserved copy. Check whether another tenant indexed the same content.

Model and orchestration. Record the provider, model, prompt, guardrail, router, and retry configuration. Find agents or background jobs that continued after the conversation ended. If the issue followed a model change, use the chatbot model rollback checklist.

Tools and queues. Inventory every available tool. Pause queued and scheduled work. Check idempotency keys, approvals, retry counters, and dead-letter queues for actions that may still execute.

Identity and credentials. Map the visitor identity, account, chatbot, worker, service account, token scopes, and delegated downstream credentials. A token that appears unused may still share authority with the compromised path.

Destinations and side effects. Ask each downstream system what changed. A 200 does not prove a refund settled, a ticket was created once, or an email reached its recipient. Reconcile destination records before restoring writes.

Preserve Evidence That Can Answer “What Changed?”

Hugging Face reported using AI-assisted analysis over more than 17,000 events to rebuild its incident timeline. Most chatbot incidents will be much smaller, but they need the same correlation discipline.

Keep an append-only event for each consequential attempt with:

• incident, trace, action, and attempt identifiers;
• actor, tenant, chatbot, worker, and credential identity;
• model, prompt, source, tool-schema, and deployment versions;
• intended operation and validated arguments;
• approval identity, approved payload hash, and expiry;
• network destination and downstream request identifier;
• response status, confirmed business effect, and retry decision;
• timestamps for request, execution, acknowledgement, and reconciliation.

Redact secret values while retaining stable fingerprints that let responders match the same credential across systems. Preserve original timestamps and hashes. If a responder transforms or annotates evidence, store the derived artifact beside the immutable original.

The AI agent audit-trail guide defines the event structure in more detail. Build it before an incident; reconstructing IDs from free-form logs during containment costs time and leaves gaps.

Be cautious when sending raw incident material to an external model. Logs can contain credentials, customer data, attacker payloads, and instructions designed to manipulate tools. Apply approved redaction, isolation, retention, and provider rules. A responder must verify every containment or disclosure decision against source evidence.

Eradicate the Path Without Destroying the Proof

Containment stops the immediate effect. Eradication removes the path that made it possible.

Patch the application-level authorization failure before tuning model behavior. Remove poisoned sources from active retrieval, validate crawler and upload boundaries, close unintended egress, reduce tool scopes, fix tenant filters, and make approvals fail closed. Rebuild compromised workers from known-good artifacts instead of trusting an in-place cleanup when runtime integrity is uncertain.

Rotate credentials from the inside out. Start with credentials proven exposed or used, then rotate identities that could mint, read, or replace them. Test the replacement on a disabled or read-only path before restoring action traffic. The credential-rotation runbook covers overlap, validation, and retirement without turning the response into a second outage.

Turn the exact incident into regression cases at multiple layers. A test should prove that the poisoned passage cannot grant authority, the tool rejects a missing approval, the downstream API rejects the old token, and a duplicate attempt cannot repeat the business effect. One green prompt test is insufficient evidence.

Recover Through a Narrower Permission Envelope

Restore service by capability, not with one global “incident resolved” switch.

1. Static answers: serve approved status and contact information without retrieval or tools.
2. Grounded read-only answers: enable clean sources and verify tenant isolation.
3. Read tools: restore narrow lookups with fresh credentials and enhanced monitoring.
4. Low-risk writes: canary one reversible action with application-level approval and idempotency.
5. High-impact writes: require incident-owner and business-owner sign-off after destination reconciliation.

For each stage, define a traffic limit, monitoring window, abort threshold, and named approver. Watch denied actions as closely as successful ones. Repeated attempts to reach a disabled tool can reveal that the original driver remains active even while damage is blocked.

Recovery ends only after queued work, customer retries, stale sessions, cached tool catalogs, and old credentials can no longer reintroduce the incident. Keep heightened monitoring long enough to cover the longest relevant token, queue, cache, and conversation lifetime.

Say What Is Known Without Guessing

Customer communication should follow verified impact, legal obligations, and the organization's incident policy. Do not wait for every technical detail before alerting the people who must protect accounts or stop downstream harm. Do not turn an early hypothesis into a claim either.

A useful update separates four things: what happened, what the team contained, what customer action is required, and when the next update will arrive. If the impact assessment is incomplete, say which systems and time window are still under review. Give support staff one approved answer and a route for affected customers; otherwise improvised replies create conflicting promises and contaminate the incident record.

After recovery, publish an internal review with contributing conditions, detection gaps, containment decisions, verified impact, remediation owners, and dated follow-ups. Avoid reducing the cause to “the AI behaved badly.” The useful causes are enforceable: a shared credential had excess scope, an approval existed only in the prompt, logs lacked action IDs, or a queue kept accepting work after the tool was disabled.

Rehearse the Failure Before Customers Find It

Run a tabletop with support, engineering, security, and the owner of each high-impact integration. Give the team an ambiguous signal, not a complete diagnosis.

Measure time to declare, time to stop the effect, evidence completeness, time to map credentials, and time to a safe partial recovery. Google DeepMind's AI-control guidance similarly treats coverage, recall, and time-to-response as distinct measures; a fast alert is weak if it watches only a fraction of tool traffic.

An AI agent incident response plan earns its value in the minutes when nobody yet knows whether the problem is a prompt, a connector, a credential, or a queue. Clear authority, durable evidence, narrow containment, and staged recovery let the team act before certainty arrives without creating a second failure.

The chatbot should return to service with less authority and better evidence than it had before the incident. That is the recovery standard worth rehearsing.

Build your chatbot for free →

No credit card required.