A strange prompt is weak evidence. Ten accounts repeating the same request, probing the same blocked action, and rotating identifiers in 15 minutes is a pattern worth reviewing.
That distinction matters now. Anthropic's September 2026 threat intelligence report describes actors splitting work across many sessions, operating account networks, and moving activity between services. OpenAI has also found that malicious activity rarely stays on one platform or one model. A website chatbot sees only its part of the operation, so chatbot abuse detection has to connect events without treating every odd customer as an attacker.
Start with this review score. It is a triage policy, not a universal benchmark. Calibrate the thresholds against your own traffic and false-positive rate.
| Signal | 0 points | 1 point | 2 points | 3 points |
|---|---|---|---|---|
| Velocity | Normal pace | Short burst from one session | Burst across several sessions | Sustained burst across accounts or routes |
| Repetition | Unrelated questions | Similar topic | Near-duplicate intent or target | Repeated sequence with minor rewrites |
| Boundary testing | No policy contact | One refusal | Repeated refusal or limit probing | Systematic variations around the same control |
| Identity churn | Stable session | Session restart | Multiple approved pseudonymous actor keys | Rapid account, token, or route rotation |
| Downstream effect | Answer only | Extra model cost | Protected data or tool attempt | Successful or partly successful external action |
Add the highest applicable value in each row. A starting policy might review 4 to 6, add friction at 7 to 9, and pause risky actions at 10 or above. Any confirmed data exposure or unauthorized write should bypass the score and enter incident response immediately.
Score behavior, not unusual wording
Attackers rewrite prompts. Legitimate customers also paste odd text, use translation tools, retry after a timeout, or share an office network. Keyword blocks and IP bans confuse those groups.
Score observable behavior instead:
- How quickly did related requests arrive?
- Did several sessions target the same account, product, coupon, file, or API action?
- Did the requests change immediately after a refusal?
- Did actor identifiers or routes rotate while the objective stayed fixed?
- Did the chatbot only answer, or did it reach a protected tool or record?
The unit of review is a cluster of related events. One event can join a cluster through time, target, request fingerprint, referral source, or action path. No single field should decide the case.
This is where abuse detection differs from chatbot scope control. Scope control decides whether one request belongs in the chatbot. Abuse detection asks whether many individually plausible requests form one operation.
Keep an investigation-ready event record
You cannot reconstruct a campaign from message text alone. Save a small event record for policy contact, refusals, rate-limit decisions, authentication changes, and tool calls. Apply your privacy and retention rules before collecting new identifiers.
| Field | What to record | Why it matters |
|---|---|---|
event_id | Unique immutable ID | Links alerts, logs, and decisions without copying the whole conversation |
occurred_at | UTC timestamp | Rebuilds sequence and measures bursts |
session_id | Random session identifier | Groups retries within one conversation |
actor_key | Approved pseudonymous account or visitor key | Connects sessions without putting raw personal data in the review queue |
intent_fingerprint | Normalized intent and protected target | Finds repeated objectives after wording changes |
policy_result | Rule ID, version, score, and decision | Shows which control fired and which version made the decision |
action_result | Tool, resource, validation result, and receipt ID | Separates harmless probing from an external effect |
route_context | Widget, page, referral class, or API path | Reveals movement between entry points |
Keep raw transcripts behind tighter access than the event index. Reviewers usually need a redacted sample, not every customer message. Set an expiration date for both the cluster and its supporting evidence.
An intent fingerprint should describe the operation without storing the exact prose. For example, six rewrites of "apply employee discount to product 8421" can map to discount_override:product_8421. A separate protected-target field can be access-controlled or tokenized if the value is sensitive.
Find clusters without reading every conversation
Start with deterministic joins. They are easier to audit than a black-box abuse score and give reviewers a reason they can inspect.
Time and target. Group events that touch the same protected target within a short window. The target may be an order, email address, coupon, account, document, or API action.
Sequence similarity. Compare the order of policy contacts. A cluster that asks for a discount, requests a staff code, then tries a checkout action is more informative than three shared keywords.
Rewrite after refusal. Record when the next request preserves the objective but changes the framing. Repeated safe neighbors are normal. Rapid variations that keep testing the same boundary deserve a higher score.
Route movement. Join attempts that move from a public widget to an API endpoint or another page while keeping the same target. OpenAI's threat reporting is a useful reminder that an operation may span several services even when each service sees only a fragment.
Shared infrastructure as supporting evidence. Network, device, or referral signals can strengthen a cluster, but shared Wi-Fi, privacy relays, corporate gateways, and accessibility tools make them poor sole identifiers. Treat them as corroboration.
Worked case: repeated discount override attempts
Imagine an online shop chatbot that can answer product questions and submit a discount request to a server-side action. The action checks eligibility before changing a cart.
At 10:02, one visitor asks for an employee discount on product 8421. The chatbot refuses because the visitor is not authenticated. Over the next 14 minutes, five new sessions ask for the same product using phrases such as "staff price," "internal promo," and "manual markdown." Three sessions present different email addresses. Two reach the discount action. Both fail server validation.
The cluster scores as follows:
- Velocity: 2 points for a burst across several sessions.
- Repetition: 2 points for one objective and one product.
- Boundary testing: 3 points for systematic rewrites after refusal.
- Identity churn: 2 points for several actor keys and email addresses.
- Downstream effect: 2 points because protected action attempts occurred but failed.
Total: 11 points. The starting policy pauses the discount action for this cluster, keeps public product answers available, and sends an evidence packet to review. It does not ban every visitor on the same network.
The reviewer discovers that all attempts came from a retail partner testing a new employee benefit before launch. That is still unauthorized use, but it is not fraud. The team gives the partner a documented test account, expires the cluster after 30 days, and adds a regression test that anonymous visitors cannot reach the discount action.
This outcome matters. A system that only blocks creates angry false positives. A system that records why it blocked can repair the legitimate path.
Use classifiers for events, then correlate them
Content classifiers can label a prompt, response, or pair. Mistral's Shieldstral announcement is a current example: the model accepts a plain-language policy question and returns a continuous safety score. That can help classify individual events and rank uncertain cases.
The classifier still sees the object you send it. It does not automatically know that eight safe-looking requests share one target, followed the same refusal sequence, or triggered the same protected action. Correlation belongs in a separate layer that works across sessions.
Use four controls with different jobs:
- Input and output moderation classifies content against a written policy.
- Session scope control redirects requests the chatbot should not handle.
- Cross-session detection groups repeated behavior and raises a review signal.
- Server authorization validates every protected read or write regardless of the model's answer.
The content moderation policy guide shows how to define categories and decision thresholds. Feed its rule IDs into the event record so repeated low-severity results can become a high-priority cluster without silently changing the moderation verdict.
Add friction where the risk lives
A cluster score should change one capability at a time. Blanket shutdowns hide which control worked and punish unrelated users.
Keep public FAQ answers available when possible. Add a short cooldown to expensive research requests. Require authentication before account-specific data. Ask for human confirmation before an irreversible write. When the cluster touches a sensitive action, pause that action while the rest of the chatbot remains useful.
Rate limits should count expensive or risky events, not only messages. Ten short prompts that each trigger a tool can cost more than one long support question. The chatbot rate-limit guide covers weighted budgets, cooldowns, and recovery paths.
Return a stable public error such as "This action needs additional verification." Keep the internal rule ID, score, and cluster ID out of the message. Detailed feedback helps a legitimate reviewer, but it also gives an attacker a map of the control.
Preserve a compact evidence packet
Every reviewed cluster should produce the same small packet:
- cluster ID, time range, and detection rule version;
- the joins that formed the cluster;
- a redacted sample of representative events;
- policy and action results, including receipt IDs;
- the containment step and its owner;
- reviewer outcome, confidence, and expiration date;
- follow-up test or product fix.
Do not label a person malicious when the evidence only shows suspicious behavior. Use outcomes such as confirmed_abuse, authorized_test, product_confusion, automation_error, or inconclusive. Those labels let you measure detector quality without turning a tentative alert into a permanent accusation.
If the cluster includes prompt injection or an unsafe tool effect, reuse the evidence and response-time rules in the prompt injection triage rubric. One incident record should connect the cluster, affected action, containment, and regression test.
Measure whether the detector helps
Track reviewed clusters, not raw alert volume. Four measures expose most problems:
- Confirmed rate equals confirmed-abuse clusters divided by reviewed clusters. A collapse may mean the detector is too broad.
- Time to containment runs from the first harmful event to the first effective control. Report the median and the slowest high-severity case.
- False-positive recovery time measures how long an authorized user waits for normal access after review.
- Recurrence rate counts confirmed clusters that reappear with the same objective after containment.
Review samples below the threshold too. If confirmed campaigns consistently score 6, a threshold of 10 provides false comfort. If legitimate shared-office traffic keeps scoring 8, reduce the weight of network corroboration and rely more on protected targets and action results.
Make repeated misuse visible before it scales
Useful chatbot abuse detection connects weak signals, limits the risky capability, and leaves enough evidence for a person to reverse the decision. The goal is not to identify every bad intention from one message. It is to catch repeated behavior before fragmented sessions look like unrelated noise.
Agentkit provides conversation logs and analytics on every plan, along with rate limiting. Use the logs for review evidence, then keep authorization checks in the system that owns each protected action.
No credit card required.
