When Local Is the Wrong Answer

Nine parts of this series argue for running models on hardware you own. This one argues the other side, using the same published numbers, because a series that only ever tells you to keep going is selling something.

Four things beat local setups reliably. Context, throughput, the ceiling on model quality, and the hours you spend. Each has a figure attached.

Context: qwen3:8b stops at 40K, Claude Opus 5 runs to 1M

The Ollama tags page publishes a context window per tag, and the numbers are smaller than the frontier by more than an order of magnitude.

Anthropic also publishes a max output of 128k tokens on Opus 5, Sonnet 5 and Fable 5, and 64k on Haiku 4.5. There is no equivalent published output ceiling on the Ollama tag pages.

The qwen3 family carries an oddity worth naming rather than smoothing over. Its own tags page lists qwen3:4b at 256K and qwen3:8b at 40K. The smaller model advertises six times the context of the larger one, and nothing on the page explains why. If you assumed context scales with model size, that single row disproves it.

The advertised number is also not what you get. Ollama's context-length page assigns 4k tokens by default under 24GB of VRAM, 32k between 24GB and 48GB, and 256k at 48GB or more. So a 40K model on a 16GB laptop starts at 4k unless you raise it yourself, and raising it costs memory out of the same pool holding the weights. The page recommends "at least 64000 tokens" for agents, search and coding, which on most consumer hardware means choosing between a large context and a large model.

None of that applies to a hosted call. You send a million tokens and the provider's problem is the provider's problem.

One machine serves one stream

llama.cpp publishes throughput for Llama-3.1-8B across its quantizations, measured on its own hardware. Generation runs from 29.17 tokens per second at F16 up to 90.01 at Q2_K_S, with Q4_K_M at 71.93. Prompt processing sits between 708.71 and 923.49 depending on the type.

Those are respectable numbers for one person typing one question. They describe one stream on one machine.

Serve five colleagues from that machine and they queue. Serve a web application and every concurrent user competes for the same weights in the same memory. Nothing in Ollama's documentation describes batching or concurrency behaviour, and there is no published guidance on requests per second for any model. The absence is itself informative: this is a tool documented for single-machine use.

Add the load cost. Ollama keeps a model resident for five minutes after use and then unloads it, so a bursty workload pays the full load repeatedly. OLLAMA_KEEP_ALIVE can pin it indefinitely, which trades that latency for permanently occupied memory.

If your use case has more than one simultaneous user, local stops being a cost saving and becomes a capacity problem you now own.

The strongest open models are the ones you cannot load

Open weights and runnable weights are different sets, and the gap between them is where most of the quality lives.

The published download sizes make the point without argument. qwen3:235b-a22b-q4_K_M is 142GB. Its q8_0 build is 250GB and its fp16 build is 470GB. gpt-oss:120b is 65GB, and OpenAI's own card describes the target as fitting "on a single 80GB GPU". gemma4:31b-it-bf16 is 63GB.

Those are the tiers people mean when they say open models have caught up. They are also unreachable on a machine with 16GB or 32GB of memory, which describes most laptops in use.

What you can run at 16GB is roughly the 8B to 12B band at four bits. That is a genuinely useful class of model in 2026, and it is not the class that appears in the benchmark posts you read about open weights.

Compare against what a hosted call costs before assuming the local one is free. Anthropic's published rates:

A Sonnet 5 introductory rate of $2 and $10 per MTok applies through 31 August 2026. Occasional use of a frontier model at those numbers is cheaper than the electricity argument suggests, and considerably cheaper than a GPU you bought to avoid it.

Setup time is a cost nobody puts on the invoice

Look at what the previous nine parts asked of you. Choose hardware against published tag sizes. Install a runtime and learn its CLI. Decode quantization labels. Discover that two documentation pages disagree about your default context. Configure an editor, reconcile a roles list that differs between two pages on the same site, and find out a JetBrains subscription is required for a model running on your own machine. Then discover that an embedding model's window is 2K and its truncation defaults to silent.

Every one of those is a real, documented obstacle from this series, not a hypothetical. They are all surmountable, and together they are a weekend.

The maintenance does not stop either. Ollama's latest tagged release is v0.32.5, published 27 July 2026, and its release note describes fixing "an MLX Metal bug that could reduce output quality for NVFP4 models". A quality regression that shipped, then got fixed. Staying current is part of the job now.

Version drift shows up in the guides too. Continue's Ollama guide states it "is based on Ollama v0.11.x and Continue v1.1.x", which is many versions behind the current release. Third-party documentation ages faster than the software, and reconciling the two is unpaid work you have signed up for.

If your honest usage is a few questions a week, that weekend never pays itself back.

Where local still wins outright

Four cases, and they are narrower than enthusiasm suggests.

• The data cannot leave. Contracts, patient records, unreleased code, anything under a policy that forbids third-party processing. This is the strongest case and it does not depend on cost or quality at all. A locally-run model at 8B is often good enough for a job a hosted model is not permitted to touch.
• There is no network. A flight, a field site, an air-gapped environment. Once weights are on disk they work with the interface down.
• Volume is high and the task is narrow. Classification, extraction, embedding a large corpus. Small models do these well, they run constantly, and per-token pricing on millions of calls is where a local machine genuinely undercuts an API.
• You want the weights pinned. A hosted model can change under you. Anthropic's docs note that every Claude model ID is a pinned snapshot, which helps, and they also publish a deprecation schedule: Claude Opus 4.1 "is deprecated and will be retired on August 5, 2026". A GGUF file on your disk does not get retired.

Notice what is missing from that list. "Cheaper" is not on it in the general case, only in the high-volume narrow case. "Better answers" is not on it at all, because the tier that would deliver them is the tier your machine cannot hold.

The setup most people should end up with

Both. Not as a compromise, as a routing decision.

Local handles the private, the repetitive and the offline. A hosted model handles the hard reasoning, the million-token document, and the tasks where being wrong is expensive. Claude, ChatGPT and the rest are not competitors to the model on your disk so much as a different tier you call when the local one is out of its depth.

Ollama's own product design points the same way. The tags pages list cloud variants next to local ones (gpt-oss:120b-cloud, gemma4:31b-cloud), and Zed's provider panel switches between http://localhost:11434 and https://ollama.com by editing one field. The vendor building the local runtime assumes you will use both.

The failure worth avoiding is the ideological one: refusing a hosted call on principle, then shipping a worse answer from a 4-bit 8B model with a 4096-token window because it ran on your own hardware. That is not privacy. It is a preference for the setup you already paid for.