FastGPTFastGPT

Dataset search citation merge

Combine, deduplicate, and rerank retrieval results from multiple knowledge base searches in a FastGPT workflow.

Dataset search citation merge — interface example

What It Does

Combines citation lists from multiple upstream Dataset Search nodes into a single datasetQuote array, deduplicates chunks by ID, re-ranks results using RRF (Reciprocal Rank Fusion), and trims the combined context to the configured Maximum Tokens budget.

Inputs, Configuration, and Outputs

ItemInternal KeyValue TypeDescription
Maximum TokenslimitNumberMaximum token budget retained after merging; lower-ranked RRF items beyond this cap are trimmed
Citation-1, Citation-2… (Dynamic)system_datasetQuoteListDatasetQuoteConnects two or more quoteQA outputs from upstream Dataset Search nodes
Dataset citationsquoteQADatasetQuoteDeduplicated, RRF-ranked, and token-trimmed citation list ready for AI Chat

Usage

AI Chat accepts a single citation list. Connect the outputs of multiple search nodes to Dataset search citation merge, then pass the combined citations to AI Chat.

This verified example supplies two demonstration lists containing three entries, including one duplicate. The output contains two citations with RRF ranking scores:

Three input citations deduplicated into two results

Open Run Details and select Dataset search citation merge to verify the merged total. This run shows 2:

Merged total and executed nodes

Example Use Cases

  1. After question classification, search different Datasets per category, then feed the merged results to a single AI Chat node. This avoids adding a separate AI Chat node to each branch.
  2. Run parallel searches using both the original user question and an expanded sub-query, then merge and deduplicate the recalled chunks with RRF before answer generation.
Edit on GitHub

Last updated

On this page

Put your business knowledge to work with AI

Import company documents and build accurate, reliable, traceable AI answers.