AI Support Pilot: How to Choose Your First Workflow

OpenAI launched Presence on July 22, 2026, with a revealing deployment rule: each agent starts with a specific job. The managed service is built around scoped knowledge, approved system access, policies, tests, and human escalation—not a vague goal to “handle support.” Anthropic's recent agent architecture guide makes the same business-value case from another direction: match technical complexity to the work instead of defaulting to the most autonomous design.

Use a six-factor score before building your AI support pilot. Rate volume, rule clarity, data readiness, reversibility, measurable outcome, and handoff readiness from 0 to 3. Start with a workflow scoring at least 14 out of 18, provided it has no hard stop such as missing identity controls or an irreversible action without approval. That scorecard is the asset to take into your first planning meeting.

The rest of this guide shows how to score candidates, reject unsafe ones, write a pilot contract, and make an expand-or-stop decision from real conversations.

Score the Workflow Before the Model

A useful pilot begins with a customer job, not a vendor or model. “Use an AI agent” is a technology decision. “Answer order-status questions for authenticated customers” is a workflow that can be counted, tested, and owned.

Score each candidate with the same rubric:

Do not hide uncertainty inside an average. If the source data deserves 1 point, record why and name who can move it to 2. A high-volume workflow with disputed policies is not a good pilot; it is a documentation project wearing an AI label.

The score is deliberately model-independent. A stronger model cannot decide which refund policy is canonical, create a missing escalation queue, or make an irreversible account action safe. If the broader operating foundation is still unclear, run the AI support readiness checklist before choosing a workflow.

Apply Four Hard Stops

A total score prevents intuition from dominating the meeting. Hard stops prevent a decent average from hiding one unacceptable weakness. Do not launch the candidate when any of these is true:

• Identity is required but not verified. Account balances, order details, private tickets, and subscription changes cannot rely on a visitor typing an email address or order number.
• The action is consequential and has no approval or reversal path. Refunds, cancellations, discounts, data deletion, and outbound messages need an explicit boundary. The chatbot approval workflow guide helps separate actions the bot may take from those a person must confirm.
• Two sources can produce different valid answers. Choose a canonical policy and an owner before training. Retrieval cannot resolve organizational disagreement.
• Failure strands the customer. The pilot needs a staffed destination, a useful context packet, and customer-facing wording for escalation and outages.

These stops do not permanently disqualify a workflow. They turn hidden prerequisites into visible work. Once identity, policy, approval, or handoff is fixed, score the candidate again.

Worked Example: Three Support Jobs Compete

Imagine a subscription retailer choosing among order status, refund eligibility, and account cancellation. The team reviews four weeks of tickets, interviews two support agents, and checks the systems each workflow would need.

Order status wins because the answer comes from a current system, success is observable, and a wrong response can be corrected without moving money or deleting access. Its 2-point handoff score also exposes a concrete prerequisite: the ticket destination exists, but the packet does not yet include order ID, lookup result, and escalation reason.

Refund eligibility scores well, but policy exceptions still need cleanup. Account cancellation loses before model testing because reversibility is zero and the workflow lacks an approval gate. The scoring meeting produces a sequence, not just a winner: fix the order-status handoff, document refund exceptions, and design confirmation plus recovery before reconsidering cancellation.

This is also a good moment to turn repeated support cases into maintained source material. The support-ticket knowledge-base workflow explains how to extract durable answers without teaching the chatbot from every historical reply.

Write a One-Page Pilot Contract

Once a workflow wins, make its boundary reviewable. A pilot contract should fit on one page and remain stable while prompts and implementation details change.

Here is the contract for the worked example:

workflow: authenticated_order_status
owner: support_operations
audience: signed-in retail customers
included:
  - report current fulfillment status
  - provide carrier tracking link when available
  - explain the published delivery window
excluded:
  - change shipping address
  - cancel or refund an order
  - estimate a date not returned by the order system
trusted_source: commerce_api
required_inputs:
  - authenticated_customer_id
  - order_id_owned_by_customer
handoff_triggers:
  - order not found after ownership check
  - carrier status unchanged for 72 hours
  - customer disputes delivered status
handoff_packet:
  - customer_id
  - order_id
  - latest_status
  - tracking_url
  - escalation_reason
success_gate:
  eligible_resolution_rate: ">= 70%"
  incorrect_answer_rate: "< 2%"
  complete_handoff_packet_rate: ">= 95%"
stop_condition:
  - another customer's order is disclosed
  - chatbot claims an unsupported refund or cancellation

The exclusions matter as much as the included tasks. They tell test writers what the chatbot must refuse or route, and they stop a successful status pilot from quietly expanding into account changes.

Treat the thresholds as business decisions, not universal benchmarks. A low-risk FAQ may tolerate a different fallback rate than a private account workflow. The point is to write the decision rule before seeing results.

Build Tests From Actual Work

Start the acceptance set with 30 to 50 historical cases from the chosen workflow. Remove personal data, preserve the facts that affected the outcome, and include ordinary cases alongside failures.

Cover at least these groups:

• normal success, including the most common wording customers use;
• missing, stale, malformed, and contradictory source data;
• wrong-account identifiers and unauthenticated visitors;
• every named policy exception;
• customer requests that begin in scope and then cross into an excluded action;
• destination outages and timeouts;
• every handoff trigger in the contract.

For each case, record the allowed outcomes. A delivered-order dispute should not pass because the response sounds empathetic; it passes when the chatbot avoids inventing a carrier explanation and sends the required packet to the correct queue.

Keep evaluation separate from model preference. The same case set should survive a prompt edit, retrieval change, or model switch. When production reveals a new failure, add a sanitized version to the set. For ongoing review after launch, use a random baseline plus targeted risk queues as described in the chatbot QA sampling guide.

Move Through Three Evidence Levels

Shadow first. Replay historical or copied live inputs without showing AI output to customers. Compare the proposed answer and route with the recorded support outcome. Shadow testing finds source and boundary problems without customer impact, but it cannot measure whether customers understand or accept the response.

Assist the support team next. Let agents see a draft, source evidence, and proposed route. Require a person to send or edit it. Record edits by reason: wrong fact, missing context, tone, policy exception, or unnecessary escalation. A high edit rate is useful evidence when reasons are structured; it is meaningless if every change is stored as “agent adjusted response.”

Release to a limited audience. Choose one channel, customer segment, or traffic percentage. Keep the contract unchanged long enough to interpret the results. Do not simultaneously add new source documents, actions, languages, and models, because a good or bad week will not tell you which change mattered.

OpenAI's July workflow adoption planner recommends deciding in advance what evidence will support continued testing, revision, or broader use. That is the discipline here: usage is an input, not proof that the workflow creates value.

Turn Pilot Numbers Into a Decision

Suppose the limited order-status pilot receives 500 conversations in one week. Of those, 320 match the contracted workflow. The chatbot correctly resolves 246, correctly hands off 61, gives 4 incorrect answers, and leaves 9 unresolved without a valid handoff.

Eligible resolution rate = 246 / 320 = 76.9%
Incorrect answer rate = 4 / 320 = 1.25%
Safe disposition rate = (246 + 61) / 320 = 95.9%

If 64 handoffs were attempted and 61 contained every required field, packet completeness is 95.3%. All three gates pass. The team should still classify the four wrong answers and nine broken outcomes before expanding. A passing average does not erase a stop condition: one cross-customer disclosure would pause the pilot even if every percentage looked excellent.

Add an operational-value check. If each correct self-service resolution avoids four minutes of support work, 246 resolutions save about 16.4 hours. Subtract review time, source maintenance, and exception handling. This prevents a technically accurate pilot from expanding when its workflow creates more operational work than it removes.

Use three decision labels:

• Expand when acceptance gates pass, no stop condition occurs, and the net operational result is positive.
• Revise when the workflow remains valuable but failures cluster around a fixable source, rule, prompt, or handoff gap.
• Stop when identity, safety, ownership, or economics fail and the remedy is not ready.

Document the evidence and the next boundary. Expansion should mean a specific change—such as moving from 10% to 30% of authenticated traffic—not “let the agent handle more.”

Make the First Boundary Reusable

The best first AI support pilot is rarely the flashiest. It is frequent enough to matter, bounded enough to test, safe enough to recover, and clear enough that a support owner can decide whether it worked. A six-factor score gets the team out of the demo mindset; the contract and acceptance gates keep the chosen workflow honest once real customers arrive.

After the first workflow passes, reuse its artifacts: the scorecard for selecting the next job, the contract format for defining scope, the test set structure for acceptance, and the decision record for expansion. That repeatable operating method is more valuable than a one-off launch.

In Agentkit, conversation logs provide the evidence for pilot review, while Q&A pairs let operators pin corrected answers before expanding the chatbot's scope.

Build your chatbot for free →

No credit card required.