The Mac Is Becoming an AI Runtime

For thirty years the interesting question about a Mac app was what it could do with the CPU, the filesystem and the window server. A language model is quietly joining that list — not as a feature you call over the network, but as something the operating system provides, the way it provides a GPU.

Two developments this year make that concrete, and the second one is the more interesting.

What Apple actually shipped#

The Foundation Models framework gave Swift developers access to Apple's on-device model. Apple describes it as "a compact, approximately 3-billion-parameter model", compressed "to 2 bits per weight (bpw) using Quantization-Aware-Training", and — the part that matters commercially — offering inference that is "free of cost and accessible with just a few lines of code".

Then at WWDC26 they went further. From the session on bringing your own model:

We previously introduced the Foundation Models framework to give you access to Apple's on-device language model. And now, we're opening up the framework to work with nearly any LLM, local or server-based.

Developers implement a LanguageModelExecutor, and the framework handles the session, the transcript and the KV cache (WWDC26 session 339).

Read that as an architecture statement rather than an API note. Apple is turning the model into a driver. One Swift interface, and which model answers becomes a deployment detail — Apple's local one, your own, or someone's server.

Why free inference changes the question#

If you have built anything on a hosted model, your design has been shaped by cost per token whether you noticed or not. Summaries get truncated. Retries get rationed. Background passes over a user's data — the ones that would make an assistant genuinely useful — never ship, because running them continuously for every user is a line item.

Take that cost to zero and those constraints dissolve. But a new one becomes obvious immediately, and it is the point of this post:

The scarce resource is no longer tokens. It is context.

A free local model that cannot see your mail, your calendar or what is on your screen is a very fast way to produce generic text. A modest model that can see the right three things beats a frontier model working blind, because most useful work is not hard reasoning — it is knowing what happened.

That reframes what is actually difficult about building assistants on a Mac:

The part that gets undersold: it is a routing layer#

Opening the framework to any provider makes it the same kind of thing MCP is for tools. MCP standardised how an assistant reaches your applications. A provider-agnostic model API standardises how an app reaches a model.

Both moves push value away from the component and toward the composition. If swapping the model is four protocol methods, then "which model" stops being a product decision and becomes a runtime one — cheap local model for the fast path, frontier model when the task earns it.

That is already how a well-built assistant should work. It is now how the platform expects you to work.

The line through the install base#

Here is the part that will be inconvenient for a lot of Mac software.

macOS Tahoe is the final version of macOS supporting Intel Macs, with only four Intel models still compatible, and its successor is Apple silicon only. Apple Intelligence never supported Intel Macs at all.

So on-device AI on the Mac is not a gradual capability curve. It is a hardware line, and it is now visible in the OS release schedule. If your product's answer to "does this work on my 2019 MacBook Pro" is a local model, the answer is no, and it will not become yes.

The honest limits#

A 3B model quantised to 2 bits per weight is not a frontier model, and Apple does not claim it is. It is good at summarising, extracting, classifying and short generation. It is not going to write your architecture document.

"Free inference" therefore means free for the tasks it can do. The useful architecture is a split: local for the fast, private, high-frequency path — dictation, extraction, triage, deciding what matters — and a frontier model, called with your own key, for the rare task that genuinely needs one. Anything else is either paying for tokens you did not need or asking a small model for judgment it cannot give.

What to do about it#

If you build for the Mac: stop treating the model as the product. It is becoming a swappable part, and the durable work is in reach, permissions, memory and knowing which context matters — the parts that do not get commoditised by the next release.

If you use a Mac: the questions worth asking a vendor are no longer about which model they use. They are where does inference happen, what can it see, and what does it remember. Those three answers determine whether an assistant is useful, and the first one determines whether your work leaves the machine.