Knowledge files, chunks, and retrieval indexes
Understand collections, MongoDB data chunks, object-storage files, and vector indexes.
This page explains current knowledge storage relationships for import and migration troubleshooting. For everyday operations, see create and maintain a knowledge base.
Relationship Between Files and Data
- A knowledge base stores model and processing configuration and contains collections.
- A collection stores source information, file identifiers, parent-folder relationships, and processing settings. Folder collections organize children; content collections contain chunks.
- Data chunks store text, Q&A content, metadata, and index associations in MongoDB's
dataset_datas, linked throughdatasetIdandcollectionId. - Original files and parsed text use object-storage services. The current file-reading path accepts S3 object keys and retains compatibility with legacy GridFS identifiers.
- Retrieval indexes use the vector backend selected for the deployment, with PG among the supported choices. Identifiers connect chunks to indexes; physical storage depends on the backend.
| Storage Layer | Primary Collections / Tables / Buckets | Stored Data | Impact of Loss or Drift |
|---|---|---|---|
| MongoDB | datasets, dataset_collections, dataset_datas, dataset_data_texts, dataset_trainings | Dataset settings, collection hierarchy, chunk text (q/a), full-text tokens (dataset_data_texts, searched with MongoDB $text for non-Milvus vector stores; Milvus BM25 uses modeldata_v2), and training queues | Knowledge bases, chunk text, and citation reader content become unavailable |
| VectorDB (PG / Milvus / OceanBase / SeekDB / OpenGauss) | modeldata / modeldata_v2 | Embedding vectors, team/dataset/collection filter scalars, and index IDs (dataId) | Semantic and hybrid search return no vector hits until indexes are rebuilt |
| S3 Object Storage (RustFS / MinIO / Cloud S3) | Private and Public Buckets | Uploaded source documents (PDF, DOCX, XLSX) and extracted document images | Source file downloads or embedded chunk images fail to load |
| Redis | BullMQ queues and cache keys | Async deletion jobs, sync schedules, and distributed locks | Background jobs pause until Redis connectivity is restored |
Earlier documentation describing PG file_id and the special manual and mark values refers to an older storage design. Investigate current deployments using the collection, chunk, and index identifiers of the installed version.
File Import Process
- Upload a file or connect an external source, select processing options, and preview parsing and segmentation.
- Create the collection with its source and processing settings.
- The backend reads, parses, and segments material according to the selected mode, persists the parsed chunks immediately with
parsed(pending index) status, and enqueues training tasks linked to the knowledge base and collection. - Training tasks call the language and embedding models required by the mode to generate vectors and retrieval indexes (transitioning
indexStatusthroughindexingtoindexed) and write the index IDs back to the corresponding chunk rows. - Inspect processing status and failures, then use a question with a known answer to verify retrieval and original-file access.
Backups and migrations must cover MongoDB, the selected vector backend, and object storage while preserving consistency among them. See Docker data migration and object storage.
Last updated