Mistral released Shieldstral on August 4 as a policy-adaptive safety classifier. Instead of forcing every application into one fixed harm taxonomy, it accepts a plain-language policy question and scores whether a prompt, response, prompt-response pair, or image violates that policy. The release makes one awkward fact hard to avoid: a flexible classifier is only useful when the product team has written a precise policy for it to enforce.
A chatbot content moderation policy should turn business risk into repeatable decisions. It must say what gets checked, what counts as a violation, what the visitor sees, when a human reviews the exchange, and which evidence the team keeps. Start with this one-page contract, then attach examples and tests to each row.
Start with a one-page moderation contract
| Policy field | Decision to record | Example |
|---|---|---|
| Scope | Which inputs and outputs are checked | Visitor text, retrieved passages, generated answers, uploaded images |
| Category | The exact behavior being classified | Directed harassment, credible threats, secret exposure |
| Boundary | Included and excluded examples | Angry criticism is allowed; targeted abuse is limited |
| Severity | Consequence if the classifier is right | Low, medium, high, critical |
| Action | What the application does | Allow, limit, replace, hand off, or stop |
| Customer copy | Reviewed response for each action | Acknowledge the issue and offer a supported next step |
| Evidence | What the review record contains | Policy version, score, decision, message ID, reviewer outcome |
| Owner | Who may change the rule | Trust lead, support lead, or legal owner |
Keep the contract short enough to review in one meeting. Put long examples in a test set, not in the core policy. A 40-page safety document that the runtime cannot map to a decision is governance theater.
Write categories as decisions, not themes
Provider categories are a useful starting point. The OpenAI moderation API reports categories such as harassment, hate, illicit activity, self-harm, sexual content, and violence. Your product still needs narrower rules because a category name does not determine the right customer experience.
Take harassment. A support chatbot may receive all of these messages:
- "Your delivery service is awful."
- "The courier was rude to me."
- "Your courier is an idiot."
- "Give me the courier's address so I can teach them a lesson."
Blocking all four would hide two legitimate complaints. Allowing all four without intervention would pass a threat and a private-data request into the normal support flow. The policy needs a boundary that distinguishes criticism of the service, a report about conduct, targeted abuse, and a credible safety risk.
Write every category with four parts: a definition, examples that violate it, nearby examples that do not, and the required action. Include business-specific categories alongside general harms. A financial chatbot may prohibit personalized investment instructions. A children's service may restrict adult topics far more tightly. A software support bot may discuss credential recovery but must never ask a visitor to paste a secret into chat.
This is where moderation becomes a product decision. The classifier can estimate whether content matches a rule. It cannot decide how much risk your business accepts or whether a safe partial answer would still help the customer.
Moderate three different objects
Checking only the visitor's message leaves two important paths unreviewed.
Input moderation. Classify what the visitor sent. Preserve legitimate intent even when the wording is angry or contains quoted harmful text. Someone reporting abuse should not receive the same response as someone issuing the abuse.
Context moderation. Check retrieved passages, tool results, and uploaded material before the model uses them. A public webpage can contain slurs in a historical quotation, a support ticket can contain credentials, and a compromised document can contain instructions aimed at the model. The policy should say whether to redact, exclude, quarantine, or permit each kind of context.
Output moderation. Check the final answer, including any tool result rendered to the visitor. Apply stricter rules to the chatbot's own words because they appear under the company's name. The brand-safety test guide covers the adversarial prompts that should probe this output gate after every model or policy change.
Record which object triggered the decision. A log entry that says only harassment: true cannot tell a reviewer whether the visitor used abusive language, a source document contained it, or the chatbot generated it.
Map every verdict to a customer action
A moderation score is not a response plan. Give the application a small set of actions that support can explain and engineering can implement.
| Action | Use when | Runtime behavior | Visitor experience |
|---|---|---|---|
| Allow | Content fits policy | Continue normally | Direct answer |
| Answer with a boundary | Part of the request is allowed | Omit the unsafe portion | Useful partial answer and a brief limit |
| Replace and recover | Generated output violates policy | Discard output and use fixed copy | Safe next step without leaked text |
| Hand off | A person must assess intent or risk | Preserve a redacted summary and route it | Clear ownership and response expectation |
| Stop and contain | Continuing could cause immediate harm | End tools, isolate evidence, alert an owner | Minimal reviewed safety copy |
Use fixed customer copy for the last three actions. Asking the same model that crossed a boundary to improvise a safer version creates a second chance to fail. Fixed copy can still include structured facts that the application has already validated, such as a support case number or opening hours.
Avoid a universal "I can't help with that" response. It hides why the request stopped, blocks harmless portions of mixed requests, and sends legitimate customers into loops. The refusal-testing guide shows how to measure false denials separately from correct refusals.
Worked case: preserve the complaint, remove the threat
Consider a delivery chatbot that receives this message:
Visitor: Your driver left my order at the wrong building. He is a worthless idiot. Give me his home address so I can deal with him.
A useful moderation receipt looks like this:
policy_version: support-safety-12 object: visitor_input categories: targeted_abuse: true credible_threat: review private_data_request: true supported_intent: missing_delivery severity: high action: hand_off retain: - order_id - delivery_timestamp redact: - requested_private_address customer_copy: delivery-safety-handoff-v3
The customer-facing response can stay plain:
I can help report the missing delivery, but I cannot provide a driver's private address. I have sent the order details and your safety concern to a delivery specialist for review. Your case number is D-1842.
The chatbot does not repeat the insult, reveal private data, or ignore the order problem. The handoff record carries the supported facts and flags the safety concern for a person. If the customer then asks when the specialist will reply, the bot should answer from the normal service policy. One high-severity turn should not lock the entire conversation into refusal mode.
Set thresholds with both probability and severity
Classifier confidence and business impact answer different questions. Google's Gemini safety settings explicitly note that harm probability does not measure harm severity. A low-probability event can still deserve a cautious action when the possible result is severe.
Set thresholds by category and action. A medium harassment score may permit the conversation with a warning in the review log. A medium score for exposed credentials may justify redaction before the model sees the text. A credible threat can require human review even when the classifier is uncertain.
Use a labeled evaluation set to choose the thresholds. Suppose 1,000 support messages include 30 real policy violations. A proposed setting blocks 27 of them, misses 3, and blocks 45 allowed complaints. A dashboard may celebrate 90 percent recall. Support will feel the 45 false blocks, and security will care which three violations passed. Break the errors down by category, severity, language, and customer action before approving the setting.
For each category, document the cost of a false block and the cost of an unsafe pass. Then choose the threshold that matches the action. Automatic account suspension needs much stronger evidence than sending a conversation to a review queue.
Test the safe neighbors around every violation
A moderation test set needs clear violations, but the neighboring allowed cases reveal whether the policy is usable. Pair each blocked prompt with examples that share the same vocabulary and have a legitimate intent:
- A threat versus a customer quoting a threat they received
- Instructions for self-harm versus a request for crisis resources
- A slur aimed at a person versus a report that quotes abusive language
- A credential pasted into chat versus a request for credential-rotation steps
- A dangerous action request versus a product safety question about that action
Add mixed requests, follow-up turns, misspellings, and the languages that produce meaningful traffic. Run the same cases against input, context, and output checks. A policy can catch harmful generation while still mishandling a quoted phrase in retrieved documentation.
Save the expected action, not an exact sentence. The wording may improve without changing the decision. The action should remain stable until an owner approves a policy update.
Version the policy like production code
Every moderation decision should include the policy version, classifier version, threshold set, object checked, returned scores, chosen action, and final customer outcome. Keep sensitive text out of broad analytics systems. Reviewers should receive the minimum redacted context needed to judge the case.
Track four operational measures: unsafe pass rate, false-block rate, successful recovery after a limit, and reviewer reversal rate. A rising reversal rate means the written boundary, examples, or threshold no longer match real traffic. Feed reversed cases back into the evaluation set.
Require review when the model, classifier, sources, supported languages, tools, or policy text changes. The chatbot change-management guide provides the wider release process. Moderation should be one named gate inside it, with an owner and rollback condition.
Good chatbot content moderation protects the customer without deleting the customer's reason for asking. Define the boundary in plain language, map it to a small set of actions, test both violations and safe neighbors, and keep a receipt for every intervention. Then a new classifier can improve enforcement without silently rewriting your business policy.
In Agentkit, conversation logs provide cases for policy review, while Q&A pairs can pin reviewed recovery language for recurring sensitive questions.
No credit card required.
