AI Agent Checkpointing: How to Recover Failed Workflows

OpenAI launched its Agents API on September 10 with durable sessions, context management, and recovery for work that can run for hours or days. The launch is a useful signal: long-running AI agents need a recovery design, not a longer timeout.

For a customer-facing agent, checkpointing means saving verified progress at boundaries where a retry could otherwise repeat a charge, message, booking, or account change. Start with this recovery ledger:

That ledger is the bookmark. The model's conversation history can explain the request, but it should not decide which side effects already happened.

A failed refund workflow, recovered safely

Consider a support agent handling a $79 subscription cancellation and refund. The workflow has five stages:

1. Confirm account and plan
2. Ask the customer to approve cancellation and a $79 refund
3. Cancel renewal in the billing system
4. Issue the refund through the payment provider
5. Email a receipt and report completion

The customer approves the request. Stage 3 succeeds. Stage 4 reaches the payment provider, which creates refund re_8421, but the agent's network connection drops before it receives the response. The worker restarts.

A naive agent rereads the chat, sees an approved refund, and calls the refund endpoint again. The customer may receive $158. A cautious but unhelpful agent refuses to continue because it cannot tell what happened.

A checkpointed run handles the ambiguity explicitly:

{
  "run_id": "run_2037",
  "workflow_version": "cancel-refund@4",
  "last_verified_checkpoint": "renewal_cancelled",
  "approved_refund_cents": 7900,
  "refund_idempotency_key": "run_2037:refund:7900",
  "refund_state": "unknown",
  "next_step": "reconcile_refund"
}

On recovery, the agent does not issue another refund. It searches the provider by idempotency key, finds re_8421, verifies the amount and account, writes the receipt, and moves the checkpoint to refund_confirmed. Only then does it send the email.

This is the central rule: resume from verified effects, not from the last sentence the model remembers writing.

Choose checkpoints around irreversible effects

Saving state after every token is expensive and tells you little about the outside world. Saving only at the end makes recovery useless. The right boundaries sit before and after a meaningful external effect.

For a support workflow, useful checkpoints might be:

• The request passed validation and has an owner.
• The customer approved an exact payload.
• A provider accepted a write under an idempotency key.
• A separate read confirmed the intended state.
• A human supplied missing information.
• The final artifact was stored and access was checked.

Keep model progress separate from business progress. "The agent drafted a refund request" is model progress. "The billing system shows renewal off" is business progress. Recovery should trust the second only after a read from the system that owns the record.

AWS's Agentic AI Lens makes the same operational point: persist state at natural boundaries and make steps idempotent so a failed run can continue from its last completed checkpoint. The principle predates agents. Agents simply make it easier to hide a distributed workflow inside a fluent conversation.

Define what each checkpoint proves

A checkpoint name should make a testable claim. step_4_done is weak because nobody knows what step 4 meant after the workflow changes. refund_confirmed is better, but it still needs a completion contract.

Store the evidence beside the checkpoint. If effect_confirmed means "refund exists," record the provider refund ID, amount, currency, customer account, and read-back time. A boolean loses the details needed to investigate a mismatch.

Workflow versions matter too. A deployment can change the meaning or order of stages while runs are paused. Bind every run to the version that created it. Resume with compatible code, run an explicit state migration, or stop for review. Quietly applying a new workflow to old state is how "resume" becomes an unplanned second execution.

Make replay safe at every write boundary

Checkpointing tells the agent where it was. Idempotency keeps the next attempt from doing the same work twice. You need both.

Give each external write a stable idempotency key derived from the run, operation, and approved payload. Reuse it when retrying the same action. Create a new key only when the customer approves a changed action.

Before replaying a timed-out write, reconcile first:

1. Query the provider by its request ID, idempotency key, or unique business reference.
2. Compare the returned record with the approved payload.
3. Mark the effect confirmed if it matches.
4. Stop for review if a conflicting effect exists.
5. Retry only when the provider proves that no prior effect exists and its API supports safe replay.

The awkward state is unknown, not failed. A timeout says the client missed a response. It does not say the server rejected the request. Treating uncertainty as failure is what creates duplicate side effects.

This complements action verification. Verification determines whether the promised result exists. Checkpointing preserves that determination so a restarted worker does not forget it.

Keep authority fresh when a run resumes

A run may wake minutes or days after approval. During the pause, the customer can lose account access, the target record can change ownership, or the approval can expire.

Recheck authorization before each consequential write and before delivering sensitive output. The checkpoint should preserve who approved what, but it should not turn an old approval into permanent access.

Ask for a new decision when:

• The amount, destination, target record, or action type changed.
• The approved limit would be exceeded.
• The workflow version changes the consequence.
• The approval expired.
• A prior effect conflicts with the requested state.

A customer who approved a $79 refund did not approve a $158 correction. A user who approved exporting one project did not approve exporting the whole account after the schema changed. The chatbot approval workflow guide shows how to bind a confirmation to the exact action instead of a vague "continue."

Compact context without rewriting facts

Long sessions eventually need context compaction. OpenAI says its Agents API compacts earlier context while preserving information needed to continue. That can help the model reason across a long run, but the compacted narrative still has a different job from the recovery ledger.

Keep four kinds of state outside the prompt:

• Provider receipts and authoritative record IDs.
• Approval payloads and authorization snapshots.
• Checkpoint transitions with timestamps and workflow versions.
• Artifacts that later stages must read without paraphrase.

Feed the model a concise view of that state when it resumes. Do not ask it to recreate IDs, amounts, or completed effects from a summary. A good resume packet says, "Refund re_8421 for USD 79.00 is confirmed; receipt email is pending." A bad one says, "The refund was probably handled earlier."

The existing async chatbot workflow pattern covers customer-visible job states, progress, cancellation, and delivery. Checkpointing sits one layer lower. It defines how the worker survives a crash without lying to the job-status interface.

Test recovery by crashing on purpose

Most recovery bugs never appear in a happy-path demo. Add failure injection at each boundary and inspect both the external system and the resumed customer message.

Use this minimum test set:

For the refund example, run the test with a payment-provider sandbox. Kill the worker immediately after the provider returns success but before the application stores the response. The resumed run passes only if one $79 refund exists, the ledger records its provider ID, and the customer receives one accurate receipt.

Count duplicate effects, unknown states older than your recovery deadline, manual interventions, recovery attempts per run, and time from restart to reconciliation. A completion rate can remain high while duplicate charges or messages quietly climb.

Give support a usable recovery screen

Operations staff should not need raw traces to decide what happens next. Show the accepted scope, current checkpoint, verified effects, unknown effects, approval status, workflow version, and the permitted recovery actions.

Avoid one generic "retry" button. Offer actions that describe their consequence:

• Reconcile provider state.
• Resume from refund_confirmed.
• Request fresh customer approval.
• Mark for engineering review.
• Close without further action.

Every manual choice should append an event with the operator, reason, and prior state. That gives the AI agent audit trail enough evidence to explain both automated and human recovery decisions.

Recovery is part of the customer promise

An agent that works for hours will eventually meet a timeout, deploy, queue retry, expired credential, or unavailable API. The quality test is not whether it avoids every interruption. It is whether it can identify the last verified effect, reconcile uncertainty, and continue without repeating harm.

Agentkit conversation logs can preserve the customer-facing result, while custom API calls and webhooks can carry stable run IDs, idempotency keys, and explicit states. Build the recovery ledger into the systems behind those actions, then let the chatbot report what the evidence proves.

Build your chatbot for free →

No credit card required.