macOS Agent Benchmarks Arrived in 2026. The Rankings Inverted.

For a long time the honest answer to "how good is this agent at running a Mac" was that nobody had measured it. Benchmarks ran on Ubuntu VMs, and everyone assumed the scores carried across.

That stopped being true in the last fourteen months, and what the new measurements found is more useful than the absence was: the scores did not carry across.

The three that now exist#

macOSWorld came first, in June 2025. 202 interactive tasks across 30 applications, "28 macOS-exclusive", with both the instructions and the OS interface localised into five languages — English, Chinese, Arabic, Japanese and Russian. Proprietary computer-use agents clear 30%; open-source models sit under 5%. It also ships a safety subset, and its authors single out deception attacks as "more general and demand[ing] immediate attention."

MacArena, June 2026: 421 manually verified tasks across 50 applications, running on Apple's native Virtualization framework on Apple Silicon. Its motivation is a direct criticism of its predecessor — macOSWorld "covers a narrow slice of first-party applications with simpler tasks, and runs on x86 virtual machines incompatible with Apple Silicon."

MacAgentBench, also June 2026: 676 tasks across 25 applications, and the design choice that makes it interesting is that roughly 60% of tasks require both GUI and CLI. That is a claim about macOS itself — it has AppleScript, an Accessibility API and a Unix shell layered in one place, so an agent measured only on clicking is being measured on the wrong thing.

The finding that matters#

MacArena's headline is not its task count. It is this:

Notably, model rankings invert between ported and macOS-native tasks, with a leading model trailing by over 26% on the MacArena subset, suggesting that macOS poses a genuinely harder environment for current GUI agents.

Read that twice. Take the same models, run them on OSWorld tasks ported to a Mac versus tasks written natively for a Mac, and the order of the leaderboard changes. A model that leads on one trails by more than 26 points on the other.

The authors' interpretation is the one to carry away: a strong score on an existing benchmark can reflect familiarity with a task distribution rather than general competence at driving a computer. Which means every "state of the art on OSWorld" claim you have read tells you less about Mac performance than it appears to, and the transfer everyone assumed does not hold.

The best Mac score belongs to a skill library, not a model#

MacAgentBench's top result is Claude Opus 4.6 running on the OpenClaw framework at 73.7% Pass@1. Impressive until you see the control: the same model, driven as a pure GUI agent on screenshots and mouse control, scores 39.2%.

The framework nearly doubles the model. And the paper is specific about which part of the framework does the work — the advantage is "primarily driven by the skill library rather than by framework design." On tasks the skill library covers, it reaches 89.4%. On tasks it does not cover, the lead largely disappears, with baseline performance around 45.6%.

That is worth being blunt about. The best published macOS agent result is substantially a measure of how many recipes somebody wrote in advance, not of how well a model reasons about an unfamiliar Mac. It generalises about as far as the recipe book does.

The number nobody puts on a slide#

Here is the one I would put first if I were choosing an assistant.

MacAgentBench reports two reliability figures. Pass@4 — solved on at least one of four attempts — is 85.2%. Pass4 — solved on all four attempts — is 58.6%.

A twenty-seven point gap between "can do this" and "does this every time."

For a demo, Pass@4 is the honest number: you are watching, and if it fails you try again. For anything unattended — the 7am brief, the follow-up that goes out on Thursday, anything you are not sitting in front of — Pass4 is the only number that means anything. An agent that gets it right three times in four is not 75% useful. It is something you have to check, which means you are still doing the task.

The scores themselves are softer than they look#

Two 2026 papers are worth knowing about before you take any number here at face value, including the ones above.

The first is an audit. BenchJack — Hao Wang, Hanchen Li, Qiuyang Mang, Alvin Cheung, Koushik Sen and Dawn Song, May 2026 — ran automated red-teaming across ten popular agent benchmarks and found 219 distinct flaws across eight categories. Their summary of what that enables is the sentence to remember:

BenchJack synthesizes reward-hacking exploits that achieve near-perfect scores on most of the benchmarks without solving a single task.

Not "inflated scores." Near-perfect, with zero tasks solved. The constructive half matters too: their patching pipeline cut hackable-task ratios from near 100% to under 10% within three iterations, and OSWorld was fully patched. These are fixable problems, and the authors fixed several of them.

The second is about honest scoring being wrong anyway. How Benchmarks Mis-Score Computer-Use Agents (July 2026) audited 150 failure-scored trajectories across five benchmarks and found 15.3% of FAIL verdicts were incorrect — 10.7% evaluator false negatives, where a valid alternative solution was rejected, and 4.7% simply broken tasks. Their conclusion is that genuine failures cluster around verification and planning rather than execution, and that a single success rate cannot explain them.

So a two-point difference between two agents on any of these leaderboards is noise. Treat the big gaps as signal and the close calls as nothing.

What is still not measured#

The gap I claimed did not exist. These do.

Nobody measures unattended, repeated automation. MacAgentBench's Pass4 is the closest anything comes, and it is a consistency metric on supervised runs, not a measure of an agent working overnight with nobody watching.

Nobody measures the permission surface. macOS gates cross-app control behind per-pair Automation grants, with Accessibility, Screen Recording and Full Disk Access as independent permissions. Benchmarks run in VMs configured to permit what the tasks need. That is the correct choice for a benchmark and it means the thing that actually stops agents on a real Mac is factored out. See ranked by blast radius.

Nobody measures memory. Every task starts cold. Nothing scores the assistant that is more useful in month three because it learned who your people are, which is the entire proposition of a personal agent.

Nobody measures whether it asked first. Scoring is end-state based. An agent that sent the email without checking and one that showed you the draft score identically. In real use that difference is most of what trust is made of, and agents will report success they did not achieve.

How to read agent claims now#

1. Ask which benchmark and which OS. "State of the art on OSWorld" is a Ubuntu result, and MacArena is direct evidence it may not transfer.
2. Ask model or scaffold. A 73.7% that becomes 39.2% without the framework is a fact about the framework.
3. Ask Pass@1 or Pass@4. And for anything unattended, ask for Pass4, which almost nobody volunteers.
4. Ask how long the tasks were. Two-minute tasks and 1.6-hour tasks produce very different numbers from the same agent.

We have not run Jarvis against any of these, and I am not going to claim a number we have not measured. MacAgentBench and MacArena are both public, which means this is now a checkable question rather than a marketing one — for us and for everyone else in the category. That is a better situation than the one I mistakenly described this morning.

Jarvis is free and open source if you would rather test it yourself than wait for a leaderboard.