FastGPTFastGPT

App Evaluation (Beta)

Evaluate FastGPT applications with test datasets, automated scoring metrics, and quality regression reports.

Starting from FastGPT v4.11.0, batch app evaluation is supported. By providing multiple QA pairs, the system automatically scores your app's responses, enabling quantitative assessment of app performance.

The system automatically scores each QA pair in the dataset and summarizes the result as an Overall Score in the evaluation list.

Create an App Evaluation

Go to the Evaluation Page

App evaluation entry and task list

App evaluation is available only in the commercial edition. In the left Studio sidebar, click App Evaluation(Beta) to open the App evaluation page, then click Create a task in the upper-right corner.

Fill in Evaluation Details

Configure task name, evaluation model, and target application

On the task creation page, provide the following:

  • Task name: A label to identify this evaluation
  • Evaluation model: The model used for scoring
  • Evaluation app: The app to be evaluated (supports simple Chat Agents and Workflows without user interaction nodes; plugins and Chat Agent V2 are not supported)

Prepare Evaluation Data

Click Click to download the CSV template for this app in the dialog to download the CSV template. The template includes these fields:

  • Global variables
  • q (question)
  • a (expected answer)
  • Chat history

Notes:

  • Maximum of 1,000 QA pairs
  • Follow the template format when filling in data

Upload the completed file and click Start the evaluation to create the task.

View Evaluation Results

Evaluation List

Evaluation list displaying progress and overall scores

The evaluation list shows all tasks with key information:

  • Progress: Current execution status
  • Executor: The user who created the task
  • Evaluation app: The app being evaluated
  • Start/End Time: Execution time range
  • Overall Score: The task's aggregate score

Use this to compare results across iterations as you improve your app.

Evaluation Details

Evaluation task details and HTTP 429 rate-limit error example

The screenshot shows an evaluation interrupted by a model HTTP 429 rate limit. Failed records display the error; resolve the rate or quota limit and retry before comparing valid scores.

Click Detail to open the Task Details dialog:

Task Overview: The top of the dialog shows overall task information, including evaluation configuration and summary statistics.

Detailed Results: The lower part of the dialog lists each QA pair with its score, showing:

  • User question
  • Standard output
  • App output
Edit on GitHub

Last updated

On this page

Put your business knowledge to work with AI

Import company documents and build accurate, reliable, traceable AI answers.