Serve a Local Model Over an OpenAI-Compatible API

Ollama already runs a server. Installing it started one, and it has been listening on port 11434 the whole time you have been using the ollama run chat. Pointing existing code at it is a one-line change to a base URL.

The compatibility is good enough that most OpenAI SDK code works untouched. It is not complete, the gaps are documented precisely, and knowing them beforehand saves you debugging a silently ignored parameter.

One base URL and an API key that does nothing

The OpenAI-compatible surface lives at http://localhost:11434/v1/. Ollama's own Python example:

from openai import OpenAI
client = OpenAI(base_url='http://localhost:11434/v1/', api_key='ollama')

That api_key value carries a comment in the documentation, and it is the whole security story in five words: "required but ignored". The SDK insists on a non-empty string. Ollama does not look at it.

The authentication page confirms it without hedging: "No authentication is required when accessing Ollama's API locally via http://localhost:11434." API keys exist in Ollama's world only for ollama.com, where you set OLLAMA_API_KEY and send Authorization: Bearer $OLLAMA_API_KEY against https://ollama.com/api.

The curl equivalent, which is the fastest way to confirm the server is answering at all:

curl -X POST http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "gpt-oss:20b", "messages": [{"role": "user", "content": "test"}]}'

Five endpoints, and the fields each one drops

Ollama publishes a per-endpoint list of what it accepts and what it does not. The unsupported columns are the useful half.

Four of those omissions will bite real code.

n is gone on both completion endpoints. Any code that asks for several candidates in one call and picks the best gets one candidate. You will need to loop and pay for each pass separately in wall-clock time.

tool_choice is gone while tools is supported. You can hand the model a tool list. You cannot force it to call a particular one, which is the mechanism a lot of structured-extraction code relies on. Use response_format instead when you need a guaranteed shape.

Image URLs are not accepted. Vision works through base64 content in the messages array, so a pipeline that passes a public image link has to fetch and encode it first.

/v1/responses is stateless. The endpoint arrived in Ollama v0.13.3, and it drops previous_response_id, conversation, truncation and stateful requests generally. Most OpenAI Responses API tutorials chain calls with previous_response_id. Against Ollama you resend the full history yourself every turn, which is the older Chat Completions pattern.

Two field semantics on /v1/models are worth knowing before you write code against the response. The docs state that "created corresponds to when the model was last modified" and "owned_by corresponds to the ollama username, defaulting to 'library'". Neither means what the OpenAI field name suggests.

A second compatibility layer speaks Anthropic Messages

Less advertised and useful if your code already targets the Anthropic SDK. The endpoint is /v1/messages on the same port, with no /v1/ suffix on the base URL:

import anthropic
client = anthropic.Anthropic(base_url='http://localhost:11434', api_key='ollama')

Supported fields include model, max_tokens, messages with text, base64 images, tool_use, tool_result and thinking blocks, plus system, stream, temperature, top_p, top_k, stop_sequences, tools and thinking. As on the OpenAI surface, tool_choice is not supported, and neither is metadata.

The limitations section is unusually candid and worth quoting in full because two of the four will change your numbers:

• "API key is accepted but not validated"
• "Token counts are approximations"
• No /v1/messages/count_tokens, no prompt caching, no batches API, no PDF support
• Base64 images only, URL images unsupported

The token-count line deserves attention if you are building cost dashboards or context-budget logic on top of a local model. The usage object comes back with input_tokens and output_tokens, and the documentation tells you not to treat them as exact.

The curl form uses Anthropic's header:

curl -X POST http://localhost:11434/v1/messages -H "Content-Type: application/json" -H "x-api-key: ollama" -d '{"model": "qwen3-coder", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'

Reaching the server from another machine

The FAQ states the default: "Ollama binds 127.0.0.1 port 11434 by default. Change the bind address with the OLLAMA_HOST environment variable."

Combine that with the authentication page and the consequence is direct. Changing the bind address puts an unauthenticated inference endpoint on your network. There is no key to add, because there is no key to check. The API key parameters on both compatibility layers are placeholders the SDKs demand.

So put something in front of it. A reverse proxy that terminates TLS and enforces a header, an SSH tunnel, or a private overlay network. Do not rely on the fact that nobody knows the port.

On Linux the variable belongs in the systemd unit, not your shell. Use sudo systemctl edit ollama, or write it into /etc/systemd/system/ollama.service.d/override.conf. A shell export has no effect on a service the init system started.

Docker gives you the same port with a cleaner boundary

Running the server in a container makes the exposure explicit rather than ambient:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

With an NVIDIA card, after the NVIDIA Container Toolkit is installed:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The -p 11434:11434 mapping is the whole security surface, and you control which interface it binds at the Docker level. The -v ollama:/root/.ollama volume is what keeps a 14GB download from evaporating with the container.

Logs come from docker logs <container-name>, and models are pulled through the container with docker exec -it ollama ollama run llama3.2.

LM Studio serves the same shape on port 1234

Worth knowing because it means "OpenAI-compatible local server" is a category rather than an Ollama feature. LM Studio exposes GET /v1/models, POST /v1/chat/completions, POST /v1/completions, POST /v1/embeddings and POST /v1/responses, with a base URL of http://localhost:1234/v1.

LM Studio's framing of the migration is the same one-line change: "You can reuse existing OpenAI clients (in Python, JS, C#, etc) by switching up the 'base URL' property".

One asymmetry in the documentation rather than in the software. Ollama publishes an explicit unsupported-fields list per endpoint. LM Studio's OpenAI-compatibility page does not enumerate unsupported parameters or limitations. That is a difference in what has been written down, and it should not be read as LM Studio supporting more. It means Ollama tells you where the edges are and LM Studio leaves you to discover them.

Verifying the swap in three calls

Before rewriting an application against a local endpoint, confirm three things in order.

1. The server answers. curl http://localhost:11434/v1/models should return your local models. An empty list means ollama ls is empty, not that the API is broken.
2. The model name matches. The model field wants the exact tag string, suffix included. qwen3:8b and qwen3 are not interchangeable unless both exist locally.
3. The context is big enough. Ollama's default is small on most machines, and an application that worked against a hosted 200K window will quietly lose the front of its prompt. Set OLLAMA_CONTEXT_LENGTH before you conclude the model is worse than the hosted one you replaced.

That third point is the one that generates the most false conclusions about local model quality. A local model given 4096 tokens against code that was written for ChatGPT or Claude is not being compared fairly, and the symptom looks like stupidity rather than truncation.