Content moderation guide
Overview of FastGPT content moderation architecture, admin policy settings, runtime checkpoints, handling strategies, and custom plugin extension.
This guide applies to FastGPT v4.18.0 and later. If you are upgrading from the legacy commercial built-in censor, also see the Content moderation upgrade and migration guide.
FastGPT implements content moderation as a dedicated moderation Plugin type managed by the fastgpt-plugin service. Platform administrators can configure moderation providers, active checkpoints, suspected-risk handling, and exception fallback policies in one console, while developers can build adapter Plugins with @fastgpt-plugin/sdk-factory to integrate internal safety gateways or third-party moderation APIs.
Architecture and Responsibilities
Content moderation separates platform policy orchestration from provider-specific API adaptation:
| Component | Responsibilities |
|---|---|
| FastGPT Main Service | Admin policy UI, AES-256-GCM secret encryption, three runtime moderation checkpoints, streaming buffer truncation, moderation records (system_moderation_records), and admin audit logging |
| FastGPT Plugin Service | Package installation and versioning for moderation Plugins, isolated worker pools (local-pool), and synchronous text checks via GET /moderation and POST /moderation/check |
| Moderation Plugin | Adapter implemented with defineModeration() that calls upstream or local rules and normalizes responses into four-tier verdict values and hits |
Key design boundaries:
- Single active Plugin: You can install and keep saved credentials for multiple moderation Plugins, but only one Active Service can be enabled at a time to avoid conflicting provider criteria.
- Synchronous text moderation: The current release focuses on synchronous text moderation (
input.content) with optional streaming session tracking; asynchronous image, audio, or video callbacks are not included. - Console access: Admin Console > Content Moderation > Moderation Policy (
/admin/moderation/policy) is available torootadministrators on deployments with an activated Commercial Edition license.
1. Usage and Admin Configuration
1.1 Install and Configure a Moderation Plugin
Sign in as a root administrator and open Admin Console > Content Moderation > Moderation Policy (/admin/moderation/policy):
- Turn on Enable Content Moderation (disabled by default).
- Add a moderation Plugin from the service panel:
- Add from Marketplace: Opens the Marketplace filtered by moderation Plugins (
/config/tool/marketplace?type=moderation), where you can install official Plugins including Baidu (baidu), Alibaba Cloud (aliyun), Tencent Cloud (tencent), Keyword (keyword), and Legacy Custom Moderation (legacy-custom-censor; see the Content moderation upgrade and migration guide for legacy protocol adaptation). - Install from Local: Upload a
.pkgarchive or.zipbundle built locally or delivered offline.
- Add from Marketplace: Opens the Marketplace filtered by moderation Plugins (
- Click Configure Parameters next to a Plugin and fill in the fields declared by its
secretSchema:- Fields marked
isSecret: trueare encrypted with AES-256-GCM on the server and masked in API responses and UI forms. - Before persisting, FastGPT sends a fixed non-sensitive test string through the Plugin to verify connectivity. If the probe returns
verdict: 'error'or fails, the save is rejected so invalid credentials do not overwrite working settings.
- Fields marked
- Activation and uninstallation rules:
- When no Plugin is enabled yet, the first configured Plugin is activated automatically.
- Click Enable on any other configured Plugin to switch the active service mutually exclusively.
- Uninstalling a Plugin is a soft delete that preserves saved credentials so reinstalling the Plugin restores your configuration. If other moderation Plugins are installed, switch away from the active Plugin before uninstalling it. Uninstalling the last moderation Plugin automatically disables content moderation.
1.2 Moderation Stages (Hooks)
Once content moderation is enabled, all three checkpoints are enabled by default and can be toggled independently:
| Stage | Trigger Timing | Scope | Behavior on Block |
|---|---|---|---|
User Input (userInput) | Before workflow scheduling or any node executes | Current user query (query / userChatInput), custom global Variables, and dynamic form inputs | Throws an error immediately and stops execution with zero model token usage; client apps can roll back optimistic chat bubbles |
Model Input (modelInput) | Before LLM invocation in AI Chat, Tool Call (toolCall), and Agent nodes | Assembled current-turn context sent to the LLM: system prompts, current user message, Dataset citations, and parsed file text from the current turn (chat history and earlier-turn files are exempt) | Fails the model node directly without calling the LLM or incurring model billing |
Final Output (finalOutput) | Before delivering text responses or tool call arguments | AI text replies, Specified Reply nodes, User Select prompts, Form Input prompts, and model-generated tool call arguments | Blocks non-streaming replies outright; buffers streaming output by interval and aborts generation on violation |
Additional runtime notes:
- In-loop tool response masking: Inside Agent and ToolCall loops, tool execution responses (including tools, child workflows, built-in Dataset search, and file reading) are moderated whenever either
modelInputorfinalOutputis enabled. Violations replace the tool output with a standardized masked message (该信息因内容审核不通过,无法返回) so the Agent can continue reasoning without aborting the entire loop. - Streaming buffer window: Instead of checking isolated tokens (which can split sensitive phrases across chunks), FastGPT accumulates streamed text and checks the cumulative text every Streaming Check Interval characters. On a block, FastGPT immediately stops emitting chunks, aborts the upstream model stream, and discards unrendered text.
1.3 Handling Strategy
Under Handling Strategy, configure three global behaviors:
| Setting | Default | Options and Impact |
|---|---|---|
| Streaming Check Interval | 100 chars/check | Positive integer (>= 1). Smaller values reduce client buffering latency at higher request volume and cost; larger values reduce moderation calls with slightly longer buffering |
| Suspected Risk | Block and Log | Action when a Plugin returns verdict: 'suspected': Block and Log (blockAndLog) or Pass and Log (passAndLog) |
| Service Exception | Pass Unmoderated Content | Fallback when a Plugin returns verdict: 'error', is unavailable, or times out: Pass Unmoderated Content (pass, fail-open) or Block Unmoderated Content (block, fail-closed — apps become unavailable if moderation fails, so choose carefully) |
Click Save at the top right to apply policy changes. Navigating to Marketplace or clicking Enable preserves unsaved draft changes on the policy page.
1.4 Online Testing and Compliance Records
- Moderation Test: When moderation is enabled with an active Plugin, the test card at the bottom of the policy page lets you submit sample text and inspect the raw four-tier verdict, normalized hit labels with scores, highlighted keywords, and error messages directly from the Plugin.
- Chat Log Details: Team members with write permission on an App can inspect per-node
moderationRecordsinside Chat Log execution details (showing stage, verdict, hit labels, keywords, and moderated content). - Platform Audit and Persistent Records: Admin lifecycle actions (install, save config, enable/switch, uninstall, and policy updates) are written to the team audit log. Every runtime moderation call is persisted asynchronously to
system_moderation_recordsand can be queried via/api/core/plugin/admin/moderation/recordsor inspected at/admin/moderation/recordsby stage and verdict.
2. Custom Moderation Plugin Development
To integrate an internal safety gateway or another third-party moderation service, you can build a custom moderation Plugin with the FastGPT Plugin toolchain:
- Setup, debugging, and publishing: Prerequisites, project layout, local debugging, and
.pkgpackaging/publishing follow the same workflow as tool Plugins—see the System Tool Development Guide and Debugging, packing, and publishing. - Scaffolding and contract: Initialize a project with
npx @fastgpt-plugin/cli create my-moderation --type moderation. ExportdefineModeration()withdefineModerationManifest(), declare credentials and provider policy IDs viasecretSchema, inspect synchronous textinput.content, and return standardizedverdict(pass/block/suspected/error) andhits. Catch upstream or network failures and returnctx.moderationError('...')instead of throwing.
When building with Claude Code, Codex, or another coding agent, copy this prompt:
Follow the official FastGPT plugin development Skill (https://raw.githubusercontent.com/labring/fastgpt-official-plugins/refs/heads/main/.agents/skills/develop-fastgpt-plugin/SKILL.md) and plugin docs to build a FastGPT `moderation` plugin for <your moderation service and API docs> using `npx @fastgpt-plugin/cli create <plugin-name> --type moderation`, including `defineModeration()`, `secretSchema`, unit tests, local debug verification, and `.pkg` packaging.Last updated