# Knowledge files, chunks, and retrieval indexes

> Understand collections, MongoDB data chunks, object-storage files, and vector indexes.

Source: https://doc.fastgpt.io/self-host/design/dataset
Last Updated: 2026-11-12

This page explains current knowledge storage relationships for import and migration troubleshooting. For everyday operations, see [create and maintain a knowledge base](https://doc.fastgpt.io/guide/dataset/lifecycle).

![Knowledge base hierarchy and underlying storage architecture](https://cdn.jsdelivr.net/gh/labring/fastgpt-img@main/diagrams/knowledge-hierarchy.en.light.svg)

## Relationship Between Files and Data

* A **knowledge base** stores model and processing configuration and contains collections.
* A **collection** stores source information, file identifiers, parent-folder relationships, and processing settings. Folder collections organize children; content collections contain chunks.
* **Data chunks** store text, Q\&A content, metadata, and index associations in MongoDB's `dataset_datas`, linked through `datasetId` and `collectionId`.
* **Original files and parsed text** use object-storage services. The current file-reading path accepts S3 object keys and retains compatibility with legacy GridFS identifiers.
* **Retrieval indexes** use the vector backend selected for the deployment, with PG among the supported choices. Identifiers connect chunks to indexes; physical storage depends on the backend.

| Storage Layer                                               | Primary Collections / Tables / Buckets                                                        | Stored Data                                                                                                                                                                                                             | Impact of Loss or Drift                                                     |
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **MongoDB**                                                 | `datasets`, `dataset_collections`, `dataset_datas`, `dataset_data_texts`, `dataset_trainings` | Dataset settings, collection hierarchy, chunk text (`q`/`a`), full-text tokens (`dataset_data_texts`, searched with MongoDB `$text` for non-Milvus vector stores; Milvus BM25 uses `modeldata_v2`), and training queues | Knowledge bases, chunk text, and citation reader content become unavailable |
| **VectorDB (PG / Milvus / OceanBase / SeekDB / OpenGauss)** | `modeldata` / `modeldata_v2`                                                                  | Embedding vectors, team/dataset/collection filter scalars, and index IDs (`dataId`)                                                                                                                                     | Semantic and hybrid search return no vector hits until indexes are rebuilt  |
| **S3 Object Storage (RustFS / MinIO / Cloud S3)**           | Private and Public Buckets                                                                    | Uploaded source documents (PDF, DOCX, XLSX) and extracted document images                                                                                                                                               | Source file downloads or embedded chunk images fail to load                 |
| **Redis**                                                   | BullMQ queues and cache keys                                                                  | Async deletion jobs, sync schedules, and distributed locks                                                                                                                                                              | Background jobs pause until Redis connectivity is restored                  |

Earlier documentation describing PG `file_id` and the special `manual` and `mark` values refers to an older storage design. Investigate current deployments using the collection, chunk, and index identifiers of the installed version.

## File Import Process

1. Upload a file or connect an external source, select processing options, and preview parsing and segmentation.
2. Create the collection with its source and processing settings.
3. The backend reads, parses, and segments material according to the selected mode, persists the parsed chunks immediately with `parsed` (pending index) status, and enqueues training tasks linked to the knowledge base and collection.
4. Training tasks call the language and embedding models required by the mode to generate vectors and retrieval indexes (transitioning `indexStatus` through `indexing` to `indexed`) and write the index IDs back to the corresponding chunk rows.
5. Inspect processing status and failures, then use a question with a known answer to verify retrieval and original-file access.

Backups and migrations must cover MongoDB, the selected vector backend, and object storage while preserving consistency among them. See [Docker data migration](https://doc.fastgpt.io/self-host/migration/docker_db) and [object storage](https://doc.fastgpt.io/self-host/config/object-storage).
