A prompt cache for local chat

Why a chat with a local model gets slower with every turn, and what SDK 3.0 does about it

For developers building a chat app on the SDK with a model that runs on the Mac (MLX). It does not apply to Apple’s on-device model or hosted models.

The short version. A model has no memory between requests. To answer your next message, your app sends it the whole conversation again, and the model reads all of it before it writes a word. That reading is the wait before the first word of the reply. In SDK 3.0 the model remembers what it already read, so a follow-up reads only what is new. It is on by default, uses memory only by default, and does not change how good the answers are.

Contents

The problem: every turn re-reads everything

Take a chat about a 14,000-token document. Turn one reads it. Turn two sends the document, the first answer and your new question, so the model reads the document again. Each tool call in a tool loop does the same. With a hosted model that is cheap and fast. On a Mac it is the slowest part: a 12-step tool loop over a long document took about 4½ minutes in SDK 2.x, with each step slower than the last. Long prompts also made memory balloon far past what the model needed, so big prompts were refused or squeezed the rest of the Mac.

What 3.0 does

What it is worth

Measured on a 64 GB Mac Studio (M5 Max). First word is the wait before the reply starts.

ScenarioSDK 2.xSDK 3.0
12-step tool loop over a 14,000-token document (Qwen3.8 27B, an open-weight model with 27 billion parameters)272 to 295 s47 s
Switching back to an earlier conversation15 s to first word2 s
Another question about the same 56,000-token document85 s to first word1.8 s
Same long system prompt after reopening the app (disk tier on)30.5 s3.4 s
A 56,000-token document: memory the process holds53 GB24 GB (4-bit)
A 230,000-token document (Qwen3-30B-A3B)RefusedRuns in 43 GB

The cache does not make the first read faster: a 211,000-token document took about 590 s to read either way. These are our numbers on one Mac; yours will differ with the model and the machine.

In a chat app

In a chat with a 23,000-token document, every follow-up read only the previous answer and the new message (a few hundred tokens), where 2.x read about 24,000 each turn. A few things make a turn slower, though never different: a speed-helper model (it turns the cache off), changing the tool set, trimming old messages, image turns, and very long previous answers.

Short one-off requests also take memory now (about 330 MB each on Qwen3.8), and can push a long chat out of the cache sooner. If your app runs many small requests next to a chat, expect the chat to be restored less often.

LocalLM Lab's AI Models panel, Prompt cache section, with the Automatic, Off, Memory and Memory + SSD choices and a status line for the selected model.
The Prompt cache controls in LocalLM Lab 3.0: Automatic is the SDK default, and the status line shows what the cache holds and how many lookups it has restored since the service started.

Using it

// Default: on, memory only. Nothing to write.
let provider = MLXModelProvider()

// Turn it off for the whole provider
let off = MLXModelProvider(promptCache: .off)

// Skip it for one session, for example a benchmark
let options = SessionOptions(bypassPromptCache: true)

// Keep saved work across relaunches (opt in, encrypted)
let persistent = MLXModelProvider(promptCache: .tiered())

// Show your users what it is doing
let status = provider.promptCacheStatus(for: modelID)

promptCacheStatus(for:) returns a state and a plain sentence you can show, such as “Faster follow-ups are on for this model (checked on this Mac on Oct 5, 2026).” MLXPromptCacheStats reports what the cache did. clear() and storedBytes on your MLXPromptCacheDisk (or MLXModelProvider.clearPromptCache()) manage the on-disk copy. Before quitting while a reply is still being written, call await MLXModelProvider.stopGenerating() so the GPU has stopped.

The disk tier

With .tiered() the saved work also goes to disk so it survives a relaunch. The files are encrypted (with AES-GCM, a standard method that also detects tampering) using a key kept in the Keychain, excluded from backups, and deleted by Clear. They hold the model’s working state from your users’ conversations, so offer your users a way to clear them. Only turn the disk tier on if your app needs the speed after a relaunch.

Where this comes from

The design follows oMLX, the Python server for MLX models, whose core ideas we studied and ported, with Apache-2.0 attribution: matching on the prompt’s actual tokens, saved restore points for models whose memory can’t simply be cut back, a memory tier plus a disk tier, and a dedicated restore point for the system prompt and tools. We differ in being a library built into each app, not a server, and in reusing whole saved conversations, which is simpler and shares less between prompts. We haven’t run oMLX head to head on the same Mac, so this is a design comparison, not a speed race.

Limits

↑ Top

New to a term (token, KV cache, MLX)? See the glossary. More: tool cost and the tool picker · SDK overview · migrating to 3.0