FastGPTFastGPT

Integrating Local Models with Ollama

Deploy local open-source LLMs and embedding models with Ollama and connect them to your self-hosted FastGPT instance.

Ollama is an open-source AI model deployment tool focused on simplifying the deployment and usage of large language models. It supports one-click download and running of various LLMs.

Installing Ollama

Ollama supports multiple installation methods, but Docker is recommended. If you install Ollama directly on your host machine, you'll need to figure out how to let the FastGPT Docker container access Ollama on the host, which can be tricky.

Use Ollama's official Docker image for one-click installation and startup (make sure Docker is installed on your machine):

docker pull ollama/ollama
docker run --rm -d --name ollama -p 11434:11434 ollama/ollama

If your FastGPT is deployed in Docker, put the Ollama container on the same network as the AI Proxy container (fastgpt-aiproxy). Starting with v4.17, AI Proxy sends model requests, and in the official Compose file AI Proxy joins only the fastgpt_aiproxy network:

docker run --rm -d --name ollama --network fastgpt_aiproxy -p 11434:11434 ollama/ollama

Host Installation

If you prefer not to use Docker, you can install directly on the host machine.

MacOS

If you're on macOS with Homebrew installed:

brew install ollama
ollama serve # Start the service after installation

Linux

On Linux, you can use a package manager. For Ubuntu:

curl https://ollama.com/install.sh | sh # Downloads and runs the official install script
ollama serve # Start the service after installation

Windows

On Windows, download the installer from the Ollama official website. Run the installer and follow the wizard. After installation, start the service in Command Prompt or PowerShell:

ollama serve # After installation, visit http://localhost:11434 in your browser to verify Ollama is running

Additional Notes

When running Ollama as a host application, configure Ollama to listen on 0.0.0.0.

1. Linux

If Ollama runs as a systemd service, edit the service file with sudo systemctl edit ollama.service. Add Environment="OLLAMA_HOST=0.0.0.0" under the [Service] section. Save and exit, then run sudo systemctl daemon-reload and sudo systemctl restart ollama to apply.

2. MacOS

Open a terminal and run launchctl setenv OLLAMA_HOST "0.0.0.0", then restart the Ollama application.

3. Windows

Open "Edit system environment variables" from the Start menu or search bar. In "System Properties", click "Environment Variables". Under "System variables", click "New" and create a variable named OLLAMA_HOST with value 0.0.0.0. Click "OK" to save, then restart Ollama from the Start menu.

Pull Model Images

After installing Ollama, no models are available locally -- you need to pull them:

# For Docker deployment, enter the container first: docker exec -it [Ollama container name] /bin/sh
ollama pull [model name]

Pull Model Images — interface example

Test Communication

After installation, verify connectivity by starting a temporary curl container on the fastgpt_aiproxy network used by AI Proxy and trying to reach Ollama:

docker run --rm --network fastgpt_aiproxy curlimages/curl http://XXX.XXX.XXX.XXX:11434  # Container: "http://[container name]:[port]", Host: "http://[host IP]:[port]" (host IP cannot be localhost)

If you see that the Ollama service is running, communication is working.

Integrating Ollama with FastGPT

1. Check Available Models

First, check which models Ollama has:

# For Docker-deployed Ollama: docker exec -it [Ollama container name] /bin/sh
ollama ls

1. Check Available Models — interface example

2. AI Proxy Integration

If you're using FastGPT's default configuration from here, AI Proxy is enabled by default.

Make sure AI Proxy can access the Ollama container. If not, refer to the installation section above -- check that the host listens on 0.0.0.0 and that the Ollama container joined the fastgpt_aiproxy network.

Run FastGPT, then go to Admin -> System Resources -> System Models -> Model provider -> New. Select Ollama as the protocol type, add your pulled model, and fill in the proxy address (http://[container name]:[port] for container deployments, or http://[host IP]:[port] for host installations using a routable host IP).

2. AI Proxy Integration — interface example

After saving and testing the channel, continue to 5. Model Addition and Usage to add and enable the model under Model configuration.

3. OneAPI Integration

This section applies when you already run a OneAPI service. Deploy OneAPI according to its official documentation and make sure AI Proxy can reach it. For new deployments, use AI Proxy Integration above or Direct Integration below.

In the OneAPI page, add a new channel with type Ollama. Enter your Ollama model name (must match exactly), then fill in the Ollama proxy address below -- default is http://address:port, without /v1. Test the channel after adding. This example uses Docker-deployed Ollama; for host-installed Ollama, use http://[host IP]:[port].

3. OneAPI Integration — interface example

After adding the channel, click Token -> Add Token, fill in the name, and configure as needed.

3. OneAPI Integration — interface example

FastGPT routes model requests through AI Proxy. Keep AIPROXY_API_ENDPOINT and AIPROXY_API_TOKEN configured. In Admin, add an OpenAI-compatible model provider with your OneAPI endpoint (http://address:port/v1) and OneAPI token, then associate the models.

Continue to 5. Model Addition and Usage to add and use models.

4. Direct Integration

To use Ollama without OneAPI, configure its OpenAI-compatible endpoint (http://address:port/v1) as a model provider in Admin. Ollama does not require authentication by default; if you enabled authentication, supply the corresponding key. FastGPT still routes requests through AI Proxy, so keep AIPROXY_API_ENDPOINT and AIPROXY_API_TOKEN configured. For a host deployment, use the host IP; for a container deployment, use an address reachable from AI Proxy.

Continue to 5. Model Addition and Usage to add and use models.

5. Model Addition and Usage

In FastGPT, go to Admin -> System Resources -> System Models -> Model configuration -> Add model. Make sure the model identifier matches the model name configured in the channel. See Model configuration for details.

5. Model Addition and Usage — interface example

5. Model Addition and Usage — interface example

Create an app in Studio and select the model you added. The model name shown is the alias you set. Note: the same model cannot be added multiple times -- the system uses the alias from the most recent addition.

5. Model Addition and Usage — interface example

6. Additional Notes

For the Ollama proxy addresses above: host-installed Ollama uses "http://[host IP]:[port]", container-deployed Ollama uses "http://[container name]:[port]".

Edit on GitHub

Last updated