Open-weight models on the Mac (MLX)

From docs/local-models-mlx.md on the 3.0.0-GA branch (updated 2026-11-16, 8bad46e). Code and examples: github.com/ancientcomputing/locallm. Overview: LocalLM SDK.

Contents

Running open-weight models on the Mac through LocalLMLabSDKInference: loading, measuring a turn, and pinning versions. Part of the model layer.

The prompt cache, which is on by default for these models in 3.0, is described in the SDK guide.

↑ Top

MLXModelProvider — run open-weight models locally (LocalLMLabSDKInference)

Use it when you want the model to run entirely on the user's Mac with no API key and no network at inference time. It's a DownloadableModelProvider, so on top of the provider basics it adds the runtime lifecycle (validate → download → capability-probe → remove) shown below.

Examples that use it: code-buddy calls validate before its first run, streams download progress, and prints residencyEventStream transitions to stderr (its README has the small-RAM-Mac walkthrough); workspace-buddy-local runs the whole model layer inside App Sandbox.

let mlx = MLXModelProvider(residentModelLimit: 1)   // 1 model resident at a time

// Before you download: is this model sane for this machine? (no weights pulled)
let check = try await mlx.validate("mlx-community/Qwen3-8B-4bit")   // PreflightResult
//   → repo reachable? MLX format? architecture supported? weight size vs this Mac's RAM?
//     each failure names its stage.

// Download, tracking progress
for try await event in mlx.download("mlx-community/Qwen3-8B-4bit") {
    if case .progress(_, _, let fraction) = event { updateBar(fraction) }
    if case .completed(let installed) = event { /* InstalledModel */ }
}

// A "Cancel" button: aborts the real transfer, not just this loop's iteration — the stream
// finishes by throwing (typically CancellationError) instead of yielding .completed. Equivalent
// to cancelling the Task driving the `for try await` above, for callers not structured that way.
mlx.cancelDownload("mlx-community/Qwen3-8B-4bit")

// Post-download smoke test — a real prompt + a trivial tool call. Authoritative for
// that model's ModelCapabilities (some downloaded models can't reliably tool-call).
let report = await mlx.capabilityProbe(installed.id)   // ModelCapabilityReport
// report.capabilities: .toolCalling, .guidedGeneration (respond(to:generating:) — run on
// languageModelSession, see "Running a turn"), .reasoningToggle
// This is authoritative. For a starting shortlist of what tool-calls and what doesn't,
// see the tested-models list — but always confirm your own model with capabilityProbe.

mlx.installed        // [InstalledModel] — what's on disk now
mlx.storageUsed      // total bytes of weights
try mlx.remove(id)   // delete weights

Tool calls on a local model. The MLX runtime parses the model's tool-call markup into calls to your tools. A call it can't accept fails the turn instead of reaching the user: an unparseable payload, a tool the session doesn't declare, or arguments that break the tool's schema throw LocalLMLabError.generation(underlying:), whose message names the reason (for example malformed_syntax) and never includes the raw model output. Before 2.1 the call markup came back as the model's answer and nothing ran, so if you showed that text, or retried on it, catch the error instead; running the turn again is often enough for a small model. A call written inside the model's reasoning (<think>…</think>) stays inert, since a model rehearsing a call hasn't decided to make it. The exception is a <think> block that opens directly on the call and never closes, which Qwen3 sometimes writes: that call runs. Since 2.1 the runtime also handles Gemma 4's array and object arguments and gpt-oss (Harmony) and Muse Glimmer tool calls; the CHANGELOG has the details.

Where the weights land — a standard Hugging Face cache (models--<org>--<name>/snapshots/<sha>/, deduped blobs/ underneath):

Context Default location
CLI / non-sandboxed app ~/.cache/huggingface/hub/
Sandboxed app (com.apple.security.app-sandbox) ~/Library/Containers/<bundle-id>/Data/Library/Caches/huggingface/hub/

The sandbox redirect is automatic. An 8B-4bit model is ≈4.5 GB and, for a sandboxed app, counts against the app's container. Relocate with MLXModelProvider(cacheDirectory:) or the HF_HUB_CACHE / HF_HOME env vars. A sandboxed app also needs com.apple.security.network.client for the download — examples/workspace-buddy-local runs the whole model layer inside App Sandbox and is the worked example.

The memory story — the whole reason MLXModelProvider isn't just "load and go":

For a production build, pin the repos you ship against: MLXModelProvider(pinnedRevisions: ["mlx-community/Qwen3-8B-4bit": "abc1234…"]) downloads that exact commit instead of tracking main. Without a pin, a repo owner moving main to different weights silently changes what your app fetches on a user's next fresh download — the same file path, a different model. A repo not listed in pinnedRevisions still tracks main as before; pin only the ones your app's behavior depends on being stable.

Supply-chain hardening for a free-text model picker — if your app lets the user type any repo id (rather than shipping a fixed list you've already pinned), pinnedRevisions can't help on its own: there's nothing to pin ahead of time. Three more pieces cover that case, all grouped into one MLXModelProvider.init parameter:

struct MyTrustPolicy: MLXModelTrustPolicy {
    func evaluate(repoID: String) async -> MLXModelTrustDecision {
        repoID.hasPrefix("mlx-community/") ? .allow : .deny(reason: "not on the allow-list")
    }
}

let mlx = MLXModelProvider(supplyChainPolicy: MLXSupplyChainPolicy(
    verification: .enabled,                              // default — see below
    trustPolicy: MyTrustPolicy(),                         // your allow/deny logic; nil = accept anything
    cacheLimits: MLXCacheLimits(maxTotalCacheBytes: 20_000_000_000)))   // refuse past 20GB

Self-pinning after a free-text download: every InstalledModel (from .completed or mlx.installed) carries resolvedRevision, the immutable commit hash the download actually resolved to, whether that came from a pin or from tracking main. Pass an MLXFilePinStore as pinStore: and the SDK persists it for you, outside the Hugging Face cache; see Pinning, updating and cleaning up model versions. If you would rather store it yourself, persist (repoID, resolvedRevision) anywhere that is not under the Hugging Face cache directory (remove(_:) deletes that whole directory, refs and all), and pass it back in pinnedRevisions before redownloading.

↑ Top

Measuring and guarding a local model's turn — usage, speed, memory, limits

Use it when you want to show how a local model performed on a turn (tokens, prompt and generation speed, memory), say why nothing has appeared yet on a long prompt, or decide how many tool rounds one turn may take. MLX models only — other providers don't report these.

Reference implementation: LocalLM Lab's Playground Benchmark panel reads all of it.

Usage and speed, per turn. Each turn's response carries the model's real token counts and an mlx.* metadata dictionary of turn totals — read it from the response's usage, or the last stream snapshot's. session.streamResponse yields only the reply text, so to read usage run the turn on languageModelSession directly (which skips the SDK's retry and tool refresh; see Running a turn):

var last: LanguageModelSession.Usage?
for try await snapshot in session.languageModelSession.streamResponse(to: prompt) { last = snapshot.usage }
let meta = last?.metadata ?? [:]
func number(_ key: String) -> Double? { meta[key].flatMap { Double("\($0)") } }

let tg = number(MLXUsageMetadataKey.tokensPerSecond)          // generation, t/s
let pp = number(MLXUsageMetadataKey.promptTokensPerSecond)    // prompt processing, t/s
let calls = number(MLXUsageMetadataKey.rounds)                // 1 + one per tool round
MLXUsageMetadataKey Value (turn total)
promptTokens, generatedTokens Tokens read and written, summed over the turn's model calls
promptSeconds, generateSeconds Time in each phase
promptTokensPerSecond (pp), tokensPerSecond (tg) Σ tokens ÷ Σ seconds — absent if no time was measured
rounds Model calls: the first, plus one per round after tool calls
stopReason stop, length or cancelled
draftTokensProposed, draftTokensAccepted, speculativeRounds With a speed helper (pairDraftModel) only
residentMemoryBytes, peakMemoryBytes, gpuMemoryLimitBytes MLX memory when the turn began (the loaded weights), its peak during the turn, and how much the GPU may use

Three things to know when reading usage:

What is it reading right now? A long prompt can take tens of seconds before the first token. MLXPrefillMonitor.active lists prompts being processed in this process — exact tokens, which model call of the turn, and since when — from the moment prefill starts until the first output:

if let reading = MLXPrefillMonitor.active.last(where: { $0.repoID == repoID }),
   Date().timeIntervalSince(reading.startedAt) > 3 {
    status = "Reading a \(reading.promptTokens.formatted())-token prompt (model call \(reading.modelCall))"
}

Poll it while a turn has produced no output (once a second is plenty); it's a cheap read.

Guards the SDK applies before every model call.

let mlx = MLXModelProvider(maxModelCallsPerTurn: 40)                        // this app's default
let agent = try lab.makeSession(route: .local, tools: tools,
                                options: .init(maxModelCallsPerTurn: .max))  // this session: no limit

Session beats provider beats 16; values below 1 count as 1.

↑ Top

Pinning, updating and cleaning up model versions

Part of supply-chain hardening for model downloads: the trust policy and verification above decide whether to fetch; pins decide which version you keep.

Why this matters. A Hugging Face repo is not a fixed, vetted artifact. Its owner can change the files behind the same name at any time, anyone can publish a repo with a look-alike name, and a download can be corrupted, oversized, or not runnable on the user's Mac. An app that pulls models at runtime inherits all of that. The pieces above (trust policy, preflight, hash verification, the cache cap) decide whether to fetch a model. Pinning decides which version you keep using, so that the content you validated is the content that runs. examples/mlx-control-room walks through each of these end to end.

A pin is one exact Hugging Face commit hash for a repo. There are two kinds, and they differ in who vouches for the version:

Where it comes from Who approved it May move to
Shipped pin pinnedRevisions in your code (build time) You, the developer Only a full commit hash you name
Captured pin MLXPinStore, written after a user's first successful download The user's own choice (trust on first use) main or a commit hash

A shipped pin always beats a captured one. A shipped pin that can't be fetched (a mismatch, or the commit is gone) fails hard and never falls back to main.

Persist captured pins with MLXFilePinStore. Pass a pin store to the provider and the SDK records (repoID, commit) after each successful download, then uses it on later downloads:

let pins = MLXFilePinStore()   // default file: mlx-pins.json, outside the HF cache
let mlx = MLXModelProvider(
    pinnedRevisions: ["mlx-community/gemma-3-270m-it-4bit": "abc1234…"],   // shipped pins
    pinStore: pins)                                                        // captured pins

Keep the pin file outside the Hugging Face cache. remove(_:) deletes the whole repo directory in the cache, so a pin stored there would vanish along with the weights it exists to protect. Pins are resolved per download, so one captured after init is honored by the same provider instance. effectivePin(for:) reports the pin in force and its source; buildTimePin(for:) reports the shipped one.

Update a pin atomically. Checking costs nothing; updating fetches first and switches last:

// What would change? Downloads nothing.
let check = try await mlx.checkPinUpdate("owner/model", to: nil)   // nil = main (captured pins only)

// Fetch and verify the new commit, then run beforeSwitch, then move the pin, then evict.
for try await event in mlx.updatePin("owner/model", to: newCommit, beforeSwitch: {
    await inference.pauseAndDrain()        // your code: stop new runs, let the current one finish
}) { /* DownloadEvent progress */ }

// Roll back: the old version is still cached, so this is instant.
for try await _ in mlx.updatePin("owner/model", to: oldCommit) {}

The order is preflight at the new commit, fetch and verify, beforeSwitch, move the pin, evict the resident model. If any step fails, the pin does not move and nothing changes. beforeSwitch is where the host pauses inference, because the SDK cannot know what else your app has running against the model. updatePins(_:beforeSwitch:) moves several repos together (a base model and its draft, say) and is all-or-nothing.

Ship a shipped-pin update without a new app release. Only a commit the developer names may move a shipped pin. The SDK never fetches a feed or decides what is "newest"; where your app learns the new commit, and how it authenticates it, is yours. Give the provider managedPinStore: MLXFileManagedPinStore(), then call updatePin(_:to:) with the commit your feed names. The saved override records the shipped pin it was based on, and applies only while that still equals this build's shipped pin. A newer app release therefore always wins over an older runtime update, and the stale override is discarded.

Clean up old versions. An update leaves the previous version on disk, which is what makes rollback instant. The SDK never prunes on its own. It gives the host the inventory:

for snap in mlx.snapshots(for: "owner/model") {
    // snap.revision, snap.isCurrent, exclusive bytes vs bytes shared with other versions
}
let freed = try mlx.removeSnapshot("owner/model", revision: oldCommit)   // refuses the current version

The cache stores each file once and shares it between versions, so removeSnapshot deletes only the files no other version uses. Exclusive bytes is what removing a version would actually free.

Pairing a second model — speed helper and specialization patch (verified live in examples/mlx-control-room, which has a Speed pair and an Adapter pair with a live on/off switch). Two ways to attach a smaller second model to a base model, both set on the provider (not SessionOptions) so they can be flipped on/off at runtime without rebuilding a session:

// Speed helper: base model verifies a small draft model's guesses in bulk instead of
// generating token-by-token. No change in output quality when it helps.
mlx.pairDraftModel(SpeculativeDecodingSpec(draftRepoID: "mlx-community/Qwen3-0.6B-4bit"),
                    with: "mlx-community/Qwen3-8B-4bit")

// Specialization patch: snaps a small LoRA/DoRA adapter onto the base model to specialize its
// style/tone/skill, instead of shipping a whole separate multi-gigabyte model.
mlx.pairAdapter(AdapterSpec(source: .huggingFace(repoID: "stbenjam/qwen3-0.6b-haiku-mlx-lora")),
                 with: "mlx-community/Qwen3-0.6B-bf16")

// Either call with `nil` clears the pairing. Both take effect on the *next* session built for
// that base repo id — safe to call at any time, including mid-run, for live A/B testing.
mlx.pairDraftModel(nil, with: "mlx-community/Qwen3-8B-4bit")
↑ Top