Integrating Local Models with Ollama
Deploy local open-source LLMs and embedding models with Ollama and connect them to your self-hosted FastGPT instance.
Ollama is an open-source AI model deployment tool focused on simplifying the deployment and usage of large language models. It supports one-click download and running of various LLMs.
Installing Ollama
Ollama supports multiple installation methods, but Docker is recommended. If you install Ollama directly on your host machine, you'll need to figure out how to let the FastGPT Docker container access Ollama on the host, which can be tricky.
Docker Installation (Recommended)
Use Ollama's official Docker image for one-click installation and startup (make sure Docker is installed on your machine):
docker pull ollama/ollama
docker run --rm -d --name ollama -p 11434:11434 ollama/ollamaIf your FastGPT is deployed in Docker, put the Ollama container on the same network as the AI Proxy container (fastgpt-aiproxy). Starting with v4.17, AI Proxy sends model requests, and in the official Compose file AI Proxy joins only the fastgpt_aiproxy network:
docker run --rm -d --name ollama --network fastgpt_aiproxy -p 11434:11434 ollama/ollamaHost Installation
If you prefer not to use Docker, you can install directly on the host machine.
MacOS
If you're on macOS with Homebrew installed:
brew install ollama
ollama serve # Start the service after installationLinux
On Linux, you can use a package manager. For Ubuntu:
curl https://ollama.com/install.sh | sh # Downloads and runs the official install script
ollama serve # Start the service after installationWindows
On Windows, download the installer from the Ollama official website. Run the installer and follow the wizard. After installation, start the service in Command Prompt or PowerShell:
ollama serve # After installation, visit http://localhost:11434 in your browser to verify Ollama is runningAdditional Notes
When running Ollama as a host application, configure Ollama to listen on 0.0.0.0.
1. Linux
If Ollama runs as a systemd service, edit the service file with sudo systemctl edit ollama.service. Add Environment="OLLAMA_HOST=0.0.0.0" under the [Service] section. Save and exit, then run sudo systemctl daemon-reload and sudo systemctl restart ollama to apply.
2. MacOS
Open a terminal and run launchctl setenv OLLAMA_HOST "0.0.0.0", then restart the Ollama application.
3. Windows
Open "Edit system environment variables" from the Start menu or search bar. In "System Properties", click "Environment Variables". Under "System variables", click "New" and create a variable named OLLAMA_HOST with value 0.0.0.0. Click "OK" to save, then restart Ollama from the Start menu.
Pull Model Images
After installing Ollama, no models are available locally -- you need to pull them:
# For Docker deployment, enter the container first: docker exec -it [Ollama container name] /bin/sh
ollama pull [model name]
Test Communication
After installation, verify connectivity by starting a temporary curl container on the fastgpt_aiproxy network used by AI Proxy and trying to reach Ollama:
docker run --rm --network fastgpt_aiproxy curlimages/curl http://XXX.XXX.XXX.XXX:11434 # Container: "http://[container name]:[port]", Host: "http://[host IP]:[port]" (host IP cannot be localhost)If you see that the Ollama service is running, communication is working.
Integrating Ollama with FastGPT
1. Check Available Models
First, check which models Ollama has:
# For Docker-deployed Ollama: docker exec -it [Ollama container name] /bin/sh
ollama ls
2. AI Proxy Integration
If you're using FastGPT's default configuration from here, AI Proxy is enabled by default.
Make sure AI Proxy can access the Ollama container. If not, refer to the installation section above -- check that the host listens on 0.0.0.0 and that the Ollama container joined the fastgpt_aiproxy network.
Run FastGPT, then go to Admin -> System Resources -> System Models -> Model provider -> New. Select Ollama as the protocol type, add your pulled model, and fill in the proxy address (http://[container name]:[port] for container deployments, or http://[host IP]:[port] for host installations using a routable host IP).

After saving and testing the channel, continue to 5. Model Addition and Usage to add and enable the model under Model configuration.
3. OneAPI Integration
This section applies when you already run a OneAPI service. Deploy OneAPI according to its official documentation and make sure AI Proxy can reach it. For new deployments, use AI Proxy Integration above or Direct Integration below.
In the OneAPI page, add a new channel with type Ollama. Enter your Ollama model name (must match exactly), then fill in the Ollama proxy address below -- default is http://address:port, without /v1. Test the channel after adding. This example uses Docker-deployed Ollama; for host-installed Ollama, use http://[host IP]:[port].

After adding the channel, click Token -> Add Token, fill in the name, and configure as needed.

FastGPT routes model requests through AI Proxy. Keep AIPROXY_API_ENDPOINT and AIPROXY_API_TOKEN configured. In Admin, add an OpenAI-compatible model provider with your OneAPI endpoint (http://address:port/v1) and OneAPI token, then associate the models.
Continue to 5. Model Addition and Usage to add and use models.
4. Direct Integration
To use Ollama without OneAPI, configure its OpenAI-compatible endpoint (http://address:port/v1) as a model provider in Admin. Ollama does not require authentication by default; if you enabled authentication, supply the corresponding key. FastGPT still routes requests through AI Proxy, so keep AIPROXY_API_ENDPOINT and AIPROXY_API_TOKEN configured. For a host deployment, use the host IP; for a container deployment, use an address reachable from AI Proxy.
Continue to 5. Model Addition and Usage to add and use models.
5. Model Addition and Usage
In FastGPT, go to Admin -> System Resources -> System Models -> Model configuration -> Add model. Make sure the model identifier matches the model name configured in the channel. See Model configuration for details.


Create an app in Studio and select the model you added. The model name shown is the alias you set. Note: the same model cannot be added multiple times -- the system uses the alias from the most recent addition.

6. Additional Notes
For the Ollama proxy addresses above: host-installed Ollama uses "http://[host IP]:[port]", container-deployed Ollama uses "http://[container name]:[port]".
Last updated