FastGPTFastGPT

Knowledge files, chunks, and retrieval indexes

Understand collections, MongoDB data chunks, object-storage files, and vector indexes.

This page explains current knowledge storage relationships for import and migration troubleshooting. For everyday operations, see create and maintain a knowledge base.

Knowledge base hierarchy and underlying storage architectureKnowledge base hierarchy and underlying storage architecture

Relationship Between Files and Data

  • A knowledge base stores model and processing configuration and contains collections.
  • A collection stores source information, file identifiers, parent-folder relationships, and processing settings. Folder collections organize children; content collections contain chunks.
  • Data chunks store text, Q&A content, metadata, and index associations in MongoDB's dataset_datas, linked through datasetId and collectionId.
  • Original files and parsed text use object-storage services. The current file-reading path accepts S3 object keys and retains compatibility with legacy GridFS identifiers.
  • Retrieval indexes use the vector backend selected for the deployment, with PG among the supported choices. Identifiers connect chunks to indexes; physical storage depends on the backend.
Storage LayerPrimary Collections / Tables / BucketsStored DataImpact of Loss or Drift
MongoDBdatasets, dataset_collections, dataset_datas, dataset_data_texts, dataset_trainingsDataset settings, collection hierarchy, chunk text (q/a), full-text tokens (dataset_data_texts, searched with MongoDB $text for non-Milvus vector stores; Milvus BM25 uses modeldata_v2), and training queuesKnowledge bases, chunk text, and citation reader content become unavailable
VectorDB (PG / Milvus / OceanBase / SeekDB / OpenGauss)modeldata / modeldata_v2Embedding vectors, team/dataset/collection filter scalars, and index IDs (dataId)Semantic and hybrid search return no vector hits until indexes are rebuilt
S3 Object Storage (RustFS / MinIO / Cloud S3)Private and Public BucketsUploaded source documents (PDF, DOCX, XLSX) and extracted document imagesSource file downloads or embedded chunk images fail to load
RedisBullMQ queues and cache keysAsync deletion jobs, sync schedules, and distributed locksBackground jobs pause until Redis connectivity is restored

Earlier documentation describing PG file_id and the special manual and mark values refers to an older storage design. Investigate current deployments using the collection, chunk, and index identifiers of the installed version.

File Import Process

  1. Upload a file or connect an external source, select processing options, and preview parsing and segmentation.
  2. Create the collection with its source and processing settings.
  3. The backend reads, parses, and segments material according to the selected mode, persists the parsed chunks immediately with parsed (pending index) status, and enqueues training tasks linked to the knowledge base and collection.
  4. Training tasks call the language and embedding models required by the mode to generate vectors and retrieval indexes (transitioning indexStatus through indexing to indexed) and write the index IDs back to the corresponding chunk rows.
  5. Inspect processing status and failures, then use a question with a known answer to verify retrieval and original-file access.

Backups and migrations must cover MongoDB, the selected vector backend, and object storage while preserving consistency among them. See Docker data migration and object storage.

Edit on GitHub

Last updated

On this page

Put your business knowledge to work with AI

Import company documents and build accurate, reliable, traceable AI answers.