Knowledge Base Full Lifecycle
Understand the three-tier asset topology, asynchronous training pipeline, status transitions, and end-to-end lifecycle maintenance.
The knowledge base is FastGPT's core repository for structured and unstructured business information. Mastering its three-tier topology, background processing workflows, and status machines enables clean data organization and reliable question-answering quality.
Three-Tier Asset Topology
The platform manages knowledge assets across three distinct layers:
- Knowledge Base: The top-level root configuring the embedding model, BM25 tokenizer settings, database persistence, and team-level collaborator permissions.
- Collection: The organizational container for assets. Content Collections store ingested files, scraped web pages, or synchronized platform records; Folder Collections establish nested hierarchical trees for batch authorization and tag categorization.
- Data Chunk: The granular unit produced by chunking algorithms. Each chunk preserves source text and index references such as index IDs and text. The vector backend stores embeddings, supporting standalone edits and index rebuilding.
Five Lifecycle Stages
Knowledge assets advance through five sequential lifecycle stages:
| Stage | Core Responsibility | System Actions and Deliverables |
|---|---|---|
| 1. Ingestion | Upload files, import spreadsheets, crawl URLs, or connect external sources | Security checks, upload to S3 storage, create collection records |
| 2. Text Parsing | Extract unstructured text and segment content by semantic boundaries | Generate pending data chunk records for worker queue consumption |
| 3. Indexing | Invoke models to generate embeddings and build inverted full-text indexes | Persist to vector database and Mongo index store; mark status Ready |
| 4. Retrieval | Respond to user queries with multi-route recall and reranking | Retrieve top candidate chunks; deliver grounded context with citations |
| 5. Maintenance | Fine-tune chunks, overwrite updated files, filter by tags, and clean up obsolete data | Trigger localized vector rebuilding or cascade deletion to keep data fresh |
Background Processing Pipeline and State Machine
Once file upload succeeds, the system processes assets asynchronously using a distributed queue (BullMQ and Mongo task records):
Processing Stages and Collection Status
Waiting, parsing and chunking, optional enhancement, and indexing describe processing stages in the diagram. Queued tasks wait for consumption; processing failures enter retry or troubleshooting paths.
The collection status uses three enum values and displays stage-specific badges in the UI:
- In progress (
running): Tasks remain in progress, displaying stage badges such as Queue, Parsing, Image analysis, Index vectorization, Q&A extraction, Automatic index generation, Image index generation, Index rebuilding, or synonym rebuilding. - Ready (
ready): Displays Ready; processing is complete and chunks are available for retrieval. - Indexing error (
error): Displays Indexing error; inspect diagnostics in the Error list modal, fix file, model balance, or network prerequisites, and retry.
Next Steps
- Import procedures: Read Document Ingestion and Chunking.
- Spreadsheet contracts: Read Structured Dataset Template Specifications.
- Search optimization: Explore Search Testing and Retrieval Parameters.
- Application integration: Follow Connecting Knowledge Bases to Applications.
Last updated
Workflow Core Mechanics and Scheduling Principles
An in-depth explanation of FastGPT's graph scheduling engine, in-edge readiness groupings, skip propagation, and interactive checkpoints.
Knowledge Base Dual-Engine Principles
Understand storage decoupling, vector retrieval, BM25 full-text indexing, hybrid search, and two-stage reranking mechanisms.