# Integrate knowledge ingestion and retrieval

> Use generated API contracts for import, processing checks, retrieval testing, and application association.

Source: https://doc.fastgpt.io/openapi/dataset
Last Updated: 2026-11-13

> **Online API Reference**: Always refer to the latest online API documentation for complete request/response schemas and endpoint availability:
>
> * Mainland China Cloud: [System OpenAPI](https://cloud.fastgpt.cn/apidoc/systemopenapi)
> * International Cloud: [System OpenAPI](https://cloud.fastgpt.io/apidoc/systemopenapi)
> * Self-Hosted Instance: `<instance>/apidoc/systemopenapi`

Prepare a personal API key and the required read/write permissions on the Dataset. See [API getting started](https://doc.fastgpt.io/openapi/intro) for authentication setup.

## Objects and identifiers

`datasetId` identifies a knowledge base and `collectionId` a collection. Content collections hold chunks; folder collections organize other collections. Chunks have retrieval indexes. The interface URL helps locate a resource; submit its identifier in the field specified by the selected operation.

## Build an ingestion path

1. Find knowledge-base creation or existing-base lookup in the generated reference. Verify creation permissions, type, and model configuration.
2. Choose a collection-creation operation for files, text, URLs, or external sources. Verify its format and processing settings.
3. Retain returned knowledge-base and collection identifiers. Use the corresponding query operation to inspect processing status and failures.
4. Once processing completes, submit a question with a known answer to the search-test operation. Inspect returned text and collection origin.
5. Associate the Dataset with an app and follow [business system app integration](https://doc.fastgpt.io/openapi/best-practices/app-integration) to verify retrieval through to an answer.

Reuse the material and expected answers from [your first knowledge-base Q\&A](https://doc.fastgpt.io/guide/getting-started/knowledge-qa).

## Example: import text and retrieve it

First create a general knowledge base with an available embedding model in the interface and copy its ID. Use the origin and key from API getting started, then set `FASTGPT_DATASET_ID`. This import creates a collection and consumes processing quota; repeating it creates another collection.

```bash
export FASTGPT_DATASET_ID='YOUR_DATASET_ID'
curl --fail-with-body "$FASTGPT_ORIGIN/api/core/dataset/collection/create/text" \
  -H "Authorization: Bearer $FASTGPT_API_KEY" \
  -H 'Content-Type: application/json' \
  --data @- <<EOF
{"datasetId":"$FASTGPT_DATASET_ID","name":"Leave policy example","text":"Employees receive 12 days of annual leave."}
EOF
```

The successful response contains `data.collectionId` and `data.results.insertLen`. After import returns, confirm processing has completed in the knowledge-base interface before running retrieval. Results should contain the 12-day annual-leave passage and collection origin. For empty results, check processing state, model permissions, and the knowledge-base ID. Inspect business `code` and `message` as well as HTTP status for failures.

```bash
curl --fail-with-body "$FASTGPT_ORIGIN/api/core/dataset/searchTest" \
  -H "Authorization: Bearer $FASTGPT_API_KEY" \
  -H 'Content-Type: application/json' \
  --data @- <<EOF
{"datasetId":"$FASTGPT_DATASET_ID","text":"How many days of annual leave do employees receive?","limit":1500}
EOF
```

## Maintenance and failures

Repeated imports, deletions, and replacements change material available to applications. Verify batch operations on a small sample before processing the remainder. Distinguish upload, parsing, segmentation, embedding, and permission failures, retaining collection IDs and errors. Check tag conditions against [current filtering](https://doc.fastgpt.io/guide/dataset/collection_tags) and the interface version.

See [third-party knowledge sources](https://doc.fastgpt.io/guide/dataset/third-party/third_dataset) for synchronization contracts. [Historical knowledge-base parameters](https://doc.fastgpt.io/openapi/legacy/dataset) preserve old fields and examples for migration comparisons.

## System OpenAPI Endpoint Reference (Knowledge Bases, Collections, Chunks, and Search)

The 24 endpoints below form the Knowledge Base OpenAPI surface and authenticate via `Authorization: Bearer <apiKey>`:

### 1. Knowledge Base Management (`dataset`)

| Method   | Path                             | Summary                                                                                     | Key Parameters (Query / Body)                                                                                                                                           |
| -------- | -------------------------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST`   | `/api/core/dataset/create`       | Create a knowledge base or folder                                                           | Body: `name` (required), `type` (`dataset`/`folder`/`websiteDataset`/`externalFile`, required), `intro` (required), `vectorModel`, `agentModel`, `vlmModel`, `parentId` |
| `POST`   | `/api/core/dataset/list`         | List accessible knowledge bases (deprecated; use `listV2`)                                  | Body: `parentId`, `type`, `searchKey`                                                                                                                                   |
| `POST`   | `/api/core/dataset/listV2`       | List paginated knowledge bases; pass `withFileCount: true` to return non-folder file counts | Body: `parentId`, `type`, `searchKey`, `datasetIds`, `withFileCount`, `offset`/`pageNum`, `pageSize`                                                                    |
| `GET`    | `/api/core/dataset/detail`       | Get knowledge base details (models, permissions, sync status)                               | Query: `id` (knowledge base ID, required)                                                                                                                               |
| `DELETE` | `/api/core/dataset/delete`       | Delete a knowledge base and child resources (owner only)                                    | Query: `id` (knowledge base ID, required)                                                                                                                               |
| `POST`   | `/api/core/dataset/batch/move`   | Batch-move knowledge bases to a target folder                                               | Body: `ids` (ID array), `parentId` (target folder ID or empty)                                                                                                          |
| `POST`   | `/api/core/dataset/batch/delete` | Batch-delete multiple knowledge bases                                                       | Body: `ids` (ID array)                                                                                                                                                  |

### 2. Collection Creation and Ingestion (`datasetCollectionCreate`)

| Method | Path                                                | Summary                                                            | Key Parameters (Body / Form)                                                                                        |
| ------ | --------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `POST` | `/api/core/dataset/collection/create`               | Create an empty collection or folder                               | Body: `datasetId` (required), `name` (required), `type` (`folder`/`virtual`, required), `parentId`, `tags`          |
| `POST` | `/api/core/dataset/collection/create/localFile`     | Upload a local file to create a collection and start training      | `multipart/form-data`: `file` (binary) + `data` (JSON string with `datasetId`, `trainingType`, `chunkSize`, `tags`) |
| `POST` | `/api/core/dataset/collection/create/text`          | Create a collection from raw text and start training               | Body: `datasetId` (required), `name` (required), `text` (required), `trainingType`, `chunkSettingMode`, `tags`      |
| `POST` | `/api/core/dataset/collection/create/link`          | Crawl a web URL to create a collection and start training          | Body: `datasetId` (required), `link` (required), `trainingType`, `metadata` (`webPageSelector`)                     |
| `POST` | `/api/core/dataset/collection/create/apiCollection` | Create a collection from an external API dataset file (deprecated) | Body: `datasetId` (required), `name` (required), `apiFileId` (required), `trainingType`                             |

### 3. Collection Query, Update, and Archive Download (`datasetCollection`)

| Method | Path                                             | Summary                                                              | Key Parameters (Query / Body)                                                                              |
| ------ | ------------------------------------------------ | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `POST` | `/api/core/dataset/collection/listV2`            | List paginated collections (supports search and `tagFilters`)        | Body: `datasetId` (required), `parentId`, `searchText`, `tagFilters`, `selectFolder`, `pageSize` (max 100) |
| `GET`  | `/api/core/dataset/collection/detail`            | Get collection details (index count, training status, file metadata) | Query: `id` (collection ID, required)                                                                      |
| `POST` | `/api/core/dataset/collection/update`            | Update collection name, parent folder, enabled state, or tags        | Body: `id` (or `datasetId` + `externalFileId`), `name`, `parentId`, `forbid`, `tags`                       |
| `POST` | `/api/core/dataset/collection/delete`            | Delete one or more collections (collection owner only)               | Query: `id` or Body: `collectionIds` (string array)                                                        |
| `POST` | `/api/core/dataset/collection/getDownloadTicket` | Request a short-lived ticket for batch collection ZIP download       | Body: `datasetId` (required), `collectionIds` (required, at least 1 collection ID)                         |
| `GET`  | `/api/core/dataset/collection/batchDownload`     | Stream a ZIP archive of collection source files using a ticket       | Query: `ticket` (required)                                                                                 |

### 4. Data Chunk Maintenance and Search Test (`datasetData` / `datasetOther`)

| Method   | Path                              | Summary                                                               | Key Parameters (Query / Body)                                                                                           |
| -------- | --------------------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `POST`   | `/api/core/dataset/data/pushData` | Push data chunks to the training queue in batch (up to 200 items)     | Body: `collectionId` (required), `data` (`{ q, a, indexes }[]` array, required), `trainingType`                         |
| `POST`   | `/api/core/dataset/data/v2/list`  | List paginated data chunks inside a collection                        | Body: `collectionId` (required), `searchText`, `offset`/`pageNum`, `pageSize`                                           |
| `GET`    | `/api/core/dataset/data/detail`   | Get a single data chunk and its vector indexes                        | Query: `id` (data chunk ID, required)                                                                                   |
| `PUT`    | `/api/core/dataset/data/update`   | Update a chunk's `q`, `a`, or `indexes` and re-embed                  | Body: `dataId` (required), `q` (required), `a`, `indexes`                                                               |
| `DELETE` | `/api/core/dataset/data/delete`   | Delete a single data chunk and its indexes                            | Query: `id` (data chunk ID, required)                                                                                   |
| `POST`   | `/api/core/dataset/searchTest`    | Run a retrieval test (semantic/full-text/hybrid, image query, rerank) | Body: `datasetId` (required), `text`, `queryImageUrls`, `limit` (max tokens), `similarity`, `searchMode`, `usingReRank` |

**Earlier section links**

[Dataset ](https://doc.fastgpt.io/openapi/legacy/dataset#dataset)

[Create Dataset ](https://doc.fastgpt.io/openapi/legacy/dataset#create-dataset)

[Get Dataset List ](https://doc.fastgpt.io/openapi/legacy/dataset#get-dataset-list)

[Get Dataset Details ](https://doc.fastgpt.io/openapi/legacy/dataset#get-dataset-details)

[Delete Dataset ](https://doc.fastgpt.io/openapi/legacy/dataset#delete-dataset)

[Collection ](https://doc.fastgpt.io/openapi/legacy/dataset#collection)

[Common Creation Parameters (Must Read) ](https://doc.fastgpt.io/openapi/legacy/dataset#common-creation-parameters-must-read)

[Create Empty Collection/Folder ](https://doc.fastgpt.io/openapi/legacy/dataset#create-empty-collectionfolder)

[Create a Text Collection ](https://doc.fastgpt.io/openapi/legacy/dataset#create-a-text-collection)

[Create a Link Collection ](https://doc.fastgpt.io/openapi/legacy/dataset#create-a-link-collection)

[Create a File Collection ](https://doc.fastgpt.io/openapi/legacy/dataset#create-a-file-collection)

[Create a Collection from an API Dataset (V1) ](https://doc.fastgpt.io/openapi/legacy/dataset#create-a-collection-from-an-api-dataset-v1)

[Create an External File Collection (Commercial) ](https://doc.fastgpt.io/openapi/legacy/dataset#create-an-external-file-collection-commercial)

[Get Collection List ](https://doc.fastgpt.io/openapi/legacy/dataset#get-collection-list)

[Get Collection Details ](https://doc.fastgpt.io/openapi/legacy/dataset#get-collection-details)

[Update Collection Info ](https://doc.fastgpt.io/openapi/legacy/dataset#update-collection-info)

[Delete Collection ](https://doc.fastgpt.io/openapi/legacy/dataset#delete-collection)

[Data ](https://doc.fastgpt.io/openapi/legacy/dataset#data)

[Data Structure ](https://doc.fastgpt.io/openapi/legacy/dataset#data-structure)

[Push Data to Processing Queue ](https://doc.fastgpt.io/openapi/legacy/dataset#push-data-to-processing-queue)

[Get Collection Data List ](https://doc.fastgpt.io/openapi/legacy/dataset#get-collection-data-list)

[Get Single Data Details ](https://doc.fastgpt.io/openapi/legacy/dataset#get-single-data-details)

[Update Single Data ](https://doc.fastgpt.io/openapi/legacy/dataset#update-single-data)

[Delete Single Data ](https://doc.fastgpt.io/openapi/legacy/dataset#delete-single-data)

[Search Test ](https://doc.fastgpt.io/openapi/legacy/dataset#search-test)
