OpenAI introduced a Data agent for ChatGPT Work on September 10. It can connect to company data, investigate changes, and build dashboards from plain-language questions. OpenAI says nearly all of its product team and more than two-thirds of its go-to-market organization use data agents for their own analysis.
That launch makes a useful capability easier to access, but chatbot teams still need to ask a sharper question than "How did support perform?" Use this weekly analysis brief instead:
Question: Which chatbot failure should we fix next? Window: 2026-09-01 through 2026-09-07 Population: 4,820 production conversations Signals: negative rating, repeated question, handoff, abandonment Unit: one customer problem, not one message Output: top five failure clusters with counts and evidence Decision: one owned fix, one test, and one review date
The brief turns conversation analysis into a decision. A dashboard can show that handoffs rose 18%. The useful result names which customer problem drove the increase, why the chatbot failed, how many conversations it affected, and what someone will change this week.
Start with a decision, not a chart
Chatbot logs can answer many questions. Mixing them together produces a pile of interesting facts and no clear next step.
Choose one decision for each analysis run. Useful questions include:
- Which missing answer caused the most avoidable handoffs?
- Which source change created a new failure cluster?
- Where did customers repeat themselves before leaving?
- Which resolved conversations reopened within seven days?
- Which action failed after the chatbot told the customer it succeeded?
Write the unit and population beside the question. If the unit is a customer problem, merge retries and repeated sessions that concern the same issue when your data allows it. If the unit is a conversation, do not count every message as an independent failure. A ten-message loop is one damaged customer interaction, not ten separate incidents.
Keep exploratory work separate from reporting. It is fine to notice an unexpected language or product pattern while investigating. Save it as the next question. Do not quietly change the denominator halfway through and present the result as if it answered the original brief.
The chatbot KPI guide defines measures such as deflection, fallback, and satisfaction. Conversation analysis works one level below those numbers. It explains what moved and points to the evidence needed for a fix.
Build one useful row per conversation
Export only the fields needed to answer the brief. More columns do not create more insight. They do create more places for personal data to leak into a spreadsheet or model prompt.
| Field | What to record | Why it matters |
|---|---|---|
conversation_id | Stable internal identifier | Lets a reviewer find the original evidence |
started_at | Timestamp in one timezone | Supports release and incident comparisons |
customer_problem | The job the customer tried to complete | Groups different wording around one need |
outcome | Resolved, handed off, abandoned, or unknown | Separates answer quality from business result |
failure_signal | Rating, repeat, refusal, tool error, or policy breach | Creates review queues without deciding the cause |
bot_version | Prompt, model, and knowledge release ID | Ties a change to its production effect |
evidence_ref | Transcript and source IDs | Makes every conclusion reviewable |
Add product, region, language, customer state, and action name only when they can change the answer. Remove names, email addresses, phone numbers, payment details, authentication tokens, and free-form fields that are not required for the analysis. Replace customer identifiers with stable pseudonyms if you need to trace repeat contacts.
Do not send the full transcript corpus into a model on the first pass. Filter out internal tests, empty sessions, duplicate retries, spam, and known monitoring traffic with deterministic rules. Keep a count of every exclusion. Then select the evidence needed for the question.
For weekly operations, retain a small random sample alongside signal-heavy conversations. Negative ratings and handoffs are efficient places to find failures, but they do not represent all traffic. The chatbot QA sampling plan shows how to keep the random baseline and targeted queues from corrupting each other's rates.
Label the symptom before naming the cause
A customer repeating a question is a signal. It is not yet a diagnosis. The chatbot may have retrieved the wrong page, missed one fact in a multi-part request, asked an unnecessary follow-up, or given the right policy in language the customer could not understand.
Review enough of each cluster to assign two labels:
What the customer experienced. Use concrete outcomes such as unsupported answer, incomplete answer, wrong refusal, repeated clarification, failed action, false success message, bad route, or silent abandonment.
What probably caused it. Use the layer someone can change, such as missing source, stale source, conflicting source, retrieval miss, instruction error, action validation, integration failure, routing rule, or unknown.
Keep "unknown" available. Analysts get into trouble when every awkward exchange must fit the first plausible story. A cluster with weak evidence should create an investigation, not a confident repair ticket.
Have a person review every high-consequence cluster and a sample of ordinary ones. Models are useful for proposing labels and grouping paraphrases. They can also invent a tidy theme, miss sarcasm, or treat a customer's incorrect assumption as proof that the chatbot failed. The transcript and governing source settle the label.
Worked example: 240 conversations become one fix
Suppose an online store analyzes 240 conversations that triggered at least one failure signal. A first clustering pass produces these groups:
| Cluster | Conversations | Confirmed failures | Customer effect | Likely cause |
|---|---|---|---|---|
| Return label never arrived | 34 | 28 | Customer cannot send item back | Action timeout reported as success |
| Delivery date repeated | 51 | 19 | Customer asks twice, then leaves | Answer omits order cutoff time |
| Discount request refused | 27 | 6 | Eligible customer reaches support | Loyalty status absent from context |
| Product comparison | 43 | 5 | Answer is generic but accurate | Thin comparison content |
| Other mixed issues | 85 | 11 | Varied | Several low-volume causes |
The largest cluster is not automatically the first fix. Delivery questions appeared 51 times, but only 19 were confirmed failures and the customer could still get help elsewhere. The return-label cluster contains 28 confirmed failures and a broken side effect. Worse, the chatbot told some customers the label had been sent.
The team opens five transcripts and checks the action log. In four, the email service timed out after the request reached it. The chatbot treated the request as complete because the integration returned no explicit failure. Retrying blindly could send two labels.
The repair ticket now has enough detail to act on:
cluster: return_label_false_success affected: 28 of 240 reviewed conversations customer_effect: return delayed after false confirmation cause: timeout has no explicit unknown-state branch fix: - check label status before retrying - show "still checking" until a label ID exists - route unresolved status to support after 60 seconds test: - simulate timeout after provider accepts request owner: commerce_integrations review_on: 2026-09-18
That is a better outcome than "reduce failed actions." It preserves the original evidence, distinguishes unknown state from failure, prevents a duplicate side effect, and gives the team a test it can rerun.
Rank by customer harm you can remove
A simple score can keep the loudest anecdote from winning every planning meeting. Use variables your team can defend:
priority = affected conversations
× confirmation rate
× customer impact
× fix confidence
Set customer impact on a short anchored scale. For example, use 1 for mild friction, 2 for a delayed answer, 4 for lost access or money, and 8 for safety, privacy, or unauthorized action. Set fix confidence between 0 and 1 based on whether the evidence points to a specific repair.
For the return-label cluster:
affected conversations = 34 confirmation rate = 28 ÷ 34 = 0.82 customer impact = 4 fix confidence = 0.90 priority = 34 × 0.82 × 4 × 0.90 = 100.4
The number has no universal meaning. Its job is to make assumptions visible. A security breach should bypass the score and trigger incident handling. A tiny but fast fix may also move ahead of a large content rewrite. Record the exception instead of bending the inputs until the desired item ranks first.
Count customers, not emotional intensity. One vivid transcript can expose a serious flaw, but repetition, confirmed impact, and repair confidence tell you how to plan the work. Preserve the vivid transcript as evidence and pair it with the denominator.
Ask the analysis tool for evidence
Whether you use SQL, a spreadsheet, or a data agent, prompts should request inspectable results. "Find insights" invites an attractive summary with unclear provenance.
Ask questions that specify the population, grouping rule, comparison, and evidence:
Using production conversations from September 1 through September 7, group repeated-question sessions by customer problem. Exclude internal tests and empty sessions. Return the ten largest clusters, conversation counts, three representative conversation IDs per cluster, and the release version. Do not infer a root cause yet.
Then investigate one cluster:
For the delivery-date cluster, compare conversations before and after release support-bot-2026-09-04.2. Count sessions where the bot omitted the 14:00 cutoff. Return the source IDs used in each answer and five examples for human review.
This sequence keeps grouping separate from diagnosis. It also gives a reviewer a way to reproduce the count. If the tool cannot return conversation IDs, query logic, exclusions, or source references, treat its answer as a lead rather than a finding.
Shared definitions matter too. OpenAI says its own data team supported broad Data agent use by creating business definitions, access rules, and safeguards for sensitive data. Chatbot teams need the same discipline on a smaller scale. Define "handoff," "resolved," "repeat," and "abandoned" once. Version those definitions when they change.
Close the loop in the next dataset
Every chosen fix needs a before window, an after window, and a regression case. Otherwise the analysis queue becomes a stream of plausible work with no proof that customer outcomes improved.
After the return-label fix ships, rerun the timeout test and inspect all matching production conversations for one week. Track false success messages and duplicate labels as separate zero-tolerance measures. Also watch total handoffs. A repair that removes false confirmation by routing every request to a person is safer, but it has not restored self-service.
Turn confirmed content failures into answer tests. The chatbot ground-truth guide explains how to preserve the expected answer, forbidden claims, and governing evidence. Use the chatbot change-management process to tie the repair to a release, stop condition, and rollback target.
The weekly review is complete when one problem has an owner, a fix, a test, and a date when fresh conversation data will judge it. Keep the remaining clusters in the evidence queue. Next week, ask the same decision-shaped question again.
In Agentkit, conversation logs and analytics supply the review pool, while Q&A pairs can pin corrected answers for failures caused by missing or ambiguous content.
No credit card required.
